DEV Community: Ben Yemini

How to Check an Agent's Diagnosis Before It Touches Production

Ben Yemini — Mon, 21 Sep 2026 20:53:24 +0000

Originally posted to causely.ai by Ben Yemini

TL;DR

When teams get ready to add agents to their on-call workflows, the hard part is deciding when to let those agents act on production versus just advise. A root cause claim only earns that trust if you can verify what the agent ruled out. Causely's causal model now exposes that reasoning directly to agents: which alternative diagnoses were considered for a symptom, what evidence ruled out those alternatives, and how far the failure would have spread if left unaddressed.

Acting vs. Advising

A recent r/kubernetes thread separated a remediation agent into two parts: an LLM proposes actions such as scaling, rolling back, cordoning, or draining, and a separate deterministic layer vetoes anything unsafe against the live cluster state before it executes. The design assumes a proposal and the decision to act on it are two different things.

The top reply doesn't argue with the architecture. It points to the actual gap: confidence in the cause and the safety to act are two different things, and they often get conflated. The suggested fix is to force the agent to explain why this diagnosis, specifically, before it hands over a proposed action. That way a human or a policy engine has something to review, not just a recommendation to accept or reject. Another engineer in the same thread pushes further: a veto needs memory. Blast-radius budgets per namespace, cooldowns, and a rule to halt and page a human if the expected effect doesn't show up. And the veto must check real cluster state itself, not trust what the agent reports about that state.

Platform teams everywhere are running into the same challenge: when is an agent's diagnostic capability good enough to let it act in production?

Why isn't "the agent is confident" the same as "the agent is right"?

An agent proposing an action and an agent explaining its diagnosis are two different things. Only one of them is checkable before anyone touches production. A confidence score is an assertion. A list of possible explanations for the symptom, with the causal chain behind each, is something a reviewer — human or automated — can inspect.

That's the same problem the thread's second engineer described: a veto that trusts the agent's self-report is a second opinion, not a check. A veto that can see what the model considered and rejected, and how large the blast radius of the winning diagnosis is, has something real to validate with.

What does it mean to ask an agent "why this and not that"?

When it comes to understanding cause and effect, most people familiar with the domain don't usually ask, "why did X happen?" They tend to ask "why X, rather than Y," a finding from explainable-AI research on contrastive explanation: an explanation is only satisfying relative to some alternative that didn't occur, even when that alternative is never stated out loud. A 2025 formalization of the problem frames it directly as "why P but not Q" and treats computing the difference between the two as the actual explanation task, not as an afterthought.

Causely's MCP server now exposes two tools designed specifically for this question. get_potential_diagnoses tells you what could explain a given symptom, not just the one the causal model selected, and shows the causal chain behind each. get_signal_potential_diagnoses runs the same question in the other direction: given an observed signal, what could explain it, and what's the causal chain connecting each candidate to that signal? Both draw on the same causal model that produces the primary diagnosis: the same reasoning, made visible instead of collapsed into a single answer.

Here is an example

Take a familiar incident: elevated latency and timeouts on a checkout service, several layers downstream from a shared database connection pool. An LLM-only agent traces this from the symptom alone. It queries telemetry, traces back from the checkout service, and finds a CPU spike on an adjacent service along the way. It diagnoses the CPU spike as the cause and restarts that service. The dashboard improves. The agent treats the incident as resolved.

The real cause, connection exhaustion on the shared database connection pool upstream, provides no direct signal at that layer, so the agent never sees it. The restart bought time by clearing the CPU spike, not by fixing the underlying problem. The incident resurfaces when the connection limit is reached again, and the agent repeats this loop.

Example of Claude explaining why this diagnosis and what else was considered but ruled out.

Calling get_signal_potential_diagnoses on the original symptom surfaces the connection-exhaustion diagnosis alongside the CPU-spike alternative, each with the causal chain showing what it explains and what it fails to account for. The CPU spike doesn't explain the downstream timeout pattern; the connection exhaustion does. That comparison is what makes the diagnosis reviewable before a remediation agent acts on it, not just the final answer.

Example of Claude visualizing the causal chain for this diagnosis

How far would this failure have spread if it wasn't caught?

A diagnosis with a small, well-understood blast radius and a diagnosis with a large, uncertain one are different risk decisions, even if a model is equally confident in both. rank_entities and get_diagnosis_observable_signals surface which entities the failure would affect and which signals to expect downstream before any action is taken.

Example of get_diagnosis_observable_signals tool call response in Claude Desktop

This is the concrete version of the veto idea from the above Kubernetes thread: a blast-radius budget is only useful if something can compute the blast radius first. Ranking entities by dependency exposure is what gives a policy engine, or a human running the on-call rotation, a number to set that budget against instead of a guess.

Where can causal explanations go further?

Today's tools answer "why this and not that" and "how far would it spread" against the current state of the causal model. The next layer is forward simulation: "what would break if this component degraded," asked before any symptom appears, not after. That requires modeling failure modes as first-class entities with their own preconditions and blast radius, rather than only reasoning from observed anomalies backward. It's a natural extension of the same causal model: the same questions, asked earlier instead of after the fact.

FAQ

What is a contrastive explanation in root cause analysis? Contrastive explanation answers "why this cause, not that one" instead of just "why did this happen." Research on explainable AI has found that people evaluate explanations relative to an alternative that didn't occur, even if that alternative is unstated. In root cause analysis, this means showing which diagnoses were considered and ruled out, not only the one that was selected.

How is this different from a deterministic safety veto layer? A safety veto checks whether a proposed action is safe to execute against live system state. Contrastive diagnosis explanation checks whether the reasoning behind the diagnosis holds up, and which alternatives were ruled out and on what evidence. The two are complementary: a veto layer is more useful with a diagnosis it can inspect, rather than a bare confidence score.

How do I use the new MCP tools to check an agent's diagnosis before it acts? Call get_potential_diagnoses or get_signal_potential_diagnoses against the symptom an agent is reasoning to see what could explain it and the causal chain behind each candidate. Pair this with rank_entities or get_diagnosis_observable_signals to see the blast radius of the leading diagnosis before approving any remediation action.

Doesn't surfacing alternative diagnoses just mean the model is less confident? No. Showing what could explain a symptom and the causal chain behind each candidate makes a diagnosis checkable, not uncertain. A model that only reports its top answer is asking to be trusted. A model that shows what else it considered and why each was ruled out is being reviewed. The second is the stronger claim.

What to do next

Read How Causal Reasoning Addresses the Limitations of LLMs in Observability for the causal model this builds on.
Read Beyond the Blast Radius for more on dependency-aware impact analysis.
Learn more about Causely MCP tools

What should trigger an autonomous agent in production?

Ben Yemini — Mon, 14 Sep 2026 21:51:17 +0000

Originally posted to causely.ai by Ben Yemini

TL;DR

Trigger an autonomous agent on a diagnosis. Anthropic's AI-native SDLC playbook closes the loop by using a deterministic script to monitor production and start a Claude session when a control band is breached. That works, but it also means the agent spends its first and most expensive phase working out what broke. A trigger that sets off a causal chain carries it further along.

Anthropic published The AI-Native SDLC Playbook last month. It walks through six stages, from plan through maintain, and describes what each one looks like when agents do the work, and people stay accountable for the decisions. Most of the attention (and value delivered) by large language models so far has gone to Build. But Stage six, Maintain, is the more interesting one, and it’s the one that turns the line into a loop. It is also the one worth reading closely if you run production.

In our last post, we set up Claude managed agents against a local Kubernetes cluster, gave them Grafana and Causely MCP tools, and worked through the authentication pattern for each. This post is about the trigger because it sets up the agent for everything it does afterward.

Stage six

Stage six closes the loop with a script that monitors production and invokes Claude when a metric falls outside its expected range. The playbook picks one metric with a rolling baseline, applies Western Electric rules so that slow drift registers alongside spikes, and selects the next step based on how far the metric strays. At 1σ, the script logs. At 2σ, Claude is invoked read-only to diagnose. At 3σ, Claude may act only by opening a PR into the review gate or by triggering a pre-approved runbook. I hope I have not lost those of you who dislike statistics.

The agent writes what it finds as an intent.md in the Stage one format, and the finding re-enters the pipeline like any other piece of work. Nobody starts it.

The detection half stays out of the model's hands. Louis Claxton, writing for Anthropic, is direct about it:

"detection stays entirely deterministic, with no model involved."

That constraint is the right one. A trigger that fired because a language model felt uneasy is not a trigger you can audit, tune, or explain to a change board.

A band breach carries one fact

A band breach tells you that a metric left its range. The affected entity, which symptoms are consequences of which, and how far the problem reaches, all must be reconstructed before the agent can decide anything. That reconstruction is the job for the diagnosis.

The other challenge with this approach is that each band watches a single metric, so you need a detection script for every failure mode you anticipate. In a cluster of dependent services, degradation spreads across many entities at once and often breaches nothing decisively on any single one. Every service looks slightly worse than usual. No script fires, and the loop never starts.

A diagnosis carries the causal chain behind it

A Causely Issue names the affected entity, designates a primary diagnosis, and exposes the chain of evidence behind it. The agent opens by explaining what could explain the symptom and why, rather than deriving that from the raw cluster state.

Causely computes this from the instrumentation you already run. The causal model encodes, ahead of time, which root causes each type of entity can have and which observable signals each one produces, on that entity and on the entities that depend on it. Faulty error handling on an RPC method, a slow consumer on a topic, a saturated connection pool on a database client: each has a known signature. A slow consumer shows up as growing queue depth on the topic and rising latency on whatever is waiting for the result. Faulty error handling shows up as a high error rate on that method and on every service that calls it.

At runtime, metrics, alerts, and events are mapped to entities and become observed signals. The model then asks, for the set of currently active signals, which diagnosis best explains them. Because the signatures are encoded in advance, this is a lookup against the model, not a search through raw telemetry. What arrives in the Issue is that diagnosis and the chain of observed signals that supports it, not a shorter list of alerts.

This inference is deterministic. The same signals on the same topology produce the same diagnosis. That matters for Stage six: the causal layer can sit on the detection side of the playbook's line without breaking the rule that no model participates in detection.

Here is an example: a Causely Issue becomes a pull request with nobody in the loop

Causely posts an Issue notification to a webhook receiver you host. The receiver turns it into the agent's first message and starts a session. The receiver is a small FastAPI app in our example repository. The Managed Agents API has no inbound webhook of its own.

Here is one session end-to-end. Payments are failing in a cluster: checkout returns errors, billing logs timeouts, and the frontend is slow. Nobody has been paged.

18:14:03. Causely raises an Issue on payment-adapter, severity Critical: calls to external-payment-api:8115 are timing out and being canceled. The receiver forwards it. The agent's first message carries the affected entity, the diagnosis, the evidence, and the Issue id, with an instruction to call get_issue_details before anything else.

18:14:05. The agent makes that call. Causely returns the causal chain. The external payment API is slow to accept charges. Payment-adapter holds the caller's request open while it waits. Checkout's 5s client timeout fires and cancels the whole chain, so the errors land three hops from the cause: 796 canceled calls on payment-adapter, 500 failed calls from billing-service, 657 timeouts on checkout, five services affected. Causely's suggested remediation is attached: add circuit-breaking in the payment-adapter so a slow dependency doesn't take checkout down with it.

18:14:19. The agent has the root cause, the evidence, and the blast radius. Sixteen seconds. It moves to the fix.

18:14:20 to 18:18:57. Everything from here happens inside the repository. The agent readspayment-adapter/main.go, finds the outbound call, edits, reviews its own diff, commits, pushes, and opens the pull request. It never goes back to Causely, Grafana, or Kubernetes. It already knows what it is fixing and why.

The pull request adds a circuit breaker and a per-call deadline around the external payment call, 87 lines in one file. The description explains why this fix, and not another: the external API is outside the repository, the cascade is inside it, and stopping the cascade restores checkout. Causely's diagnosis and evidence are in the PR body for the reviewer.

Four minutes and fifty-four seconds from Issue to pull request. Seventeen seconds of that was diagnosis. The first human involvement is the review.

Example managed agent being triggered by Causely Issue notification

Under a threshold trigger, the same session would have opened on "checkout error rate is high" and spent its first minutes listing services, pulling error rates, and reading logs to discover that the payment-adapter was where the cancellations originated and that the external API was why. Here, that work was done deterministically by the causal model before the session started. The agent's tokens are spent on the code fix.

FAQ

What is a control band in the AI-native SDLC?

A control band is the expected range for a production metric, defined by a deterministic script rather than by a model. Anthropic's AI-native SDLC playbook sets bands from a rolling baseline and tiers the response by how far the metric strays: log at 1σ, invoke an agent read-only to diagnose at 2σ, allow a pull request or a pre-approved runbook at 3σ.

What is the difference between triggering an agent on an alert and triggering it on a diagnosis?

An alert reports that one signal left its range. A diagnosis names the affected entity and exposes the causal chain that explains the symptom. An agent, triggered by an alert, reconstructs the chain from the raw cluster state before it can act. An agent, triggered by a diagnosis, starts from that diagnosis and spends its budget on the fix.

How do I trigger a Claude managed agent from an external system?

Host a receiver yourself. The Managed Agents API supports cron and manual deployment triggers, and its webhook helper handles outbound notifications about session state, so there is no inbound webhook you can point a monitoring tool at. Run a small HTTP service that authenticates the payload, turns it into the agent's first message, and calls the sessions API.

How do I keep an autonomous agent from investigating everything?

Filter at the source. Send only Critical and High severity notifications, and send Issues rather than individual defects, so that one incident produces one investigation instead of several. Every session costs tokens and runs without supervision, so the severity filter does double duty as a cost control and a noise control.

If the agent starts from a diagnosis, who checks the diagnosis? The reviewer on the pull request, using the causal chain the diagnosis carries. Causely exposes what could explain the symptom and the evidence for each candidate, so a person or a second agent can ask why this diagnosis rather than an alternative before any change is made. Branch protection keeps the agent from approving its own work.

What to do next

Clone the example repository and point a Causely Issue notification at a managed agent. The Kubernetes half runs against a local kind cluster, so you can see a session start from a notification before you connect anything real.

Next Claude Managed Agents post: multiple scenarios run with and without causal context, measuring tokens, time to diagnosis, and whether the pull request the agent opened fixed the fault.

How Open-Inspect and Causely close the loop on autonomous remediation

Ben Yemini — Thu, 27 Aug 2026 20:36:29 +0000

Originally posted to causely.ai by Enlin Xu

TL;DR

Open-Inspect gives a background agent somewhere to run: a full development environment, triggered by an event, with no human at the keyboard. Causely gives that agent a causal diagnosis instead of a pile of telemetry to interpret. The Cloud & Platform Operations team at Fountain wired both together for proactive production ops, and the pattern is now documented in a runnable example.

On June 12, 2026, we ran a live session with Cole Murray, creator of Open-Inspect, and Martin Roberts, Cloud & Platform Operations lead at Fountain. The topic was proactive remediation: agents that fix issues before anyone notices, not agents that wait for a page. Watch the recording for Martin's walkthrough of the production setup. This post provides more detail on the mechanics behind this setup and includes a link to a working example.

What does Open-Inspect run?

Open-Inspect is an open-source background coding agent, built by Cole Murray and modeled on Ramp's internal Inspect tool. It gives an agent a full development environment (Node.js, Python, git, browser automation, VS Code) and lets that agent pick up work from a webhook, a cron schedule, a Slack message, or a monitoring alert, then run unattended until the task is done.

Two important aspects of Open-Inspect are relevant to ops use cases. First, sessions persist: an agent can run for minutes or hours and be re-triggered later on the same thread, which fits an investigation that doesn't resolve in one shot. Second, the security model is single-tenant by design. Every user in a deployment shares the same GitHub App installation and repository access. That works for a trusted internal team where everyone already has access to the same repos. Everyone sharing that deployment shares the same GitHub App access. Organizations with multiple teams that need separate boundaries can run separate deployments, one per team, rather than pooling everyone into one.

Open-Inspect decides where an agent runs and what it's allowed to touch. It has no opinion on why and when to act. This is the context it needs to receive from another source.

What does Causely add to a background agent?

An agent with a working development environment and repository access still must be told what to act on. It's similar to the point we made in our blog about Claude Managed Agents: an agent that can reach every system in your environment still isn't one you'd trust to act on a guess.

The Causely MCP server gives your agent a specific, mechanistic diagnosis, such as Postgres idle-in-transaction accumulation or Redis cache miss storm, that is distinct from the symptoms it produces. A peer-reviewed study we published showed that agents working from Causely's causal context reached a diagnosis 2-3x faster than agents reasoning over raw telemetry.

How does Causely make agents faster?

The reason agents leveraging Causely diagnose problems faster is that they never need to rebuild the environment's dependency structure at query time. Not only dependencies between entities in the environment, but also the symptoms that may be observed and the potential causes that could produce those symptoms given the topology at that moment in time. Read the full benchmark study for the methodology here.

How does the integration work?

Open-Inspect answers where and how an agent can act. Causely answers when and what it should act on. An agent with both can efficiently move from an issue to a verified fix without a human bridging the gap.

Causely notification triggers the agent to Open PR

Causely runs continuously over the environment, ingesting telemetry locally and maintaining a semantic understanding of the system. When it identifies an issue, a webhook fires carrying both the issue and the diagnosis. That notification routes to the specific background agent with repo access for that environment. The agent calls get_issue_details to retrieve the causal context behind it and other Causely MCP tools, like get_logs, if it needs more evidence. Then it opens a PR.

The agent never starts from a raw alert. The webhook already carries a diagnosis, not a symptom. get_issue_details and get_logs add depth to that diagnosis. They don't build it from scratch. The output is a PR, and a human reviews the fix to verify the change before it ships.

Fountain runs this exact sequence: webhook, get_issue_details, get_logs if needed, PR. An engineer reviews and merges from there. Martin described fixing issues before the traditional alert would have fired, let alone before any end-user escalation:

"An agent works right away. It receives the request and, within seconds, identifies the root cause with help from the Causely MCP tool. Within minutes it has the pull request ready. That is much, much faster than how any ops team can function while working on the problem".
— Martin Roberts, Cloud & Platform Operations lead at Fountain

What to do next

Watch the full webinar for Martin's production walkthrough and the live Q&A.
Read the causely-oss example and adapt the trigger pattern to your own alerts.
If you're building a custom agent instead, start with Causely MCP Skills for the tool-selection logic Open-Inspect's example already uses.

FAQ

What's the difference between Open-Inspect and Causely? Open-Inspect is a background agent execution system: it decides where an agent runs and what it's allowed to touch. Causely is a causal reasoning system: it determines what issues need to be acted on and when. Open-Inspect provides the environment; Causely provides the diagnosis. Neither replaces the other.

Can a background agent remediate an issue without a causal model? It can work for specific events and runbooks that you have defined in advance, and it can work for localized, code-level problems. But in environments with many dependencies and complex topology, a background agent seeking to remediate an issue without a causal model is just as likely to make the situation worse as it is to fix the root cause. A causal model ensures the agent acts on the cause of the observed anomalies, rather than on the symptoms themselves.

Is Open-Inspect safe to deploy across multiple teams? Open-Inspect is single-tenant by design: everyone in a deployment shares the same GitHub App installation and repository scope. It's built for a trusted internal team with shared repository access, not for isolating multiple organizations or teams with different permissions within a single deployment.

How do I trigger a background agent from a production alert? Open-Inspect sessions can be triggered by webhooks, so a monitoring alert can start a session directly. The session then queries Causely for a diagnosis before taking any action, rather than starting from the raw alert payload. The causely-oss example documents this trigger pattern end-to-end.

Is access enough? Auth patterns for Claude Managed Agents

Ben Yemini — Mon, 17 Aug 2026 19:52:38 +0000

Originally posted to causely.ai by Ben Yemini

TL; DR

Background agents run independently and can be triggered by events, making them a good fit for on-call work. Claude Managed Agents are one example, and MCP tunnels give them access to the environment they're making decisions over, reaching it, and authenticating. But access is just the starting point. An agent that can reach and log into every system still isn't automatically one you'd trust to run on-call.

Why do background agents fit on-call work?

Background agents differ from interactive assistants in one key way: they run on their own schedule, triggered by an event (e.g., a page, an alert) rather than by someone typing a prompt. This difference is what makes them an ideal fit for on-call ops work, where nobody has to be sitting at a keyboard when performance degrades.

Background agents persist. They pick up an event, run for as long as the investigation takes, from seconds to hours, and can be triggered again later on the same thread. Cole Murray, the creator of Open-Inspect, walks through this distinction in detail, and it's worth watching: the architecture decisions that make sense for a chat assistant don't automatically transfer to something that runs unattended.

Claude Managed Agents provide a similar instantiation: their sessions persist in the cloud, execute over time, and can be triggered by automated events. Anthropic's own walkthrough of managed agents covers the mechanics and shows how to build an SRE incident response agent.

For on-call work, this approach is helpful in two ways. For reactive workflows, it ensures that whoever prompts the agent has the same setup and that the agent has the necessary access to assist with the investigation. More importantly, for proactive workflows, the agent can be triggered by events and start and end the investigation before an on-call engineer gets out of bed.

Recently, at Code with Claude in London on May 19, 2026, Anthropic shipped two features that improve the performance of these agents when running them against production systems. Self-hosted sandboxes (public beta) and MCP tunnels (research preview). Together, they let a cloud-run agent reach a private Kubernetes cluster or an internal Prometheus instance without opening an inbound port, closing a gap that had been blocking some on-call use cases on security review alone. These welcomed improvements make this approach much more practical. It also surfaces the next key question: once the agent has access, does it have the context it needs to resolve the problem?

Reaching your infrastructure is (mostly) a solved problem

A managed agent runs in Anthropic's cloud; your cluster, your metrics, and your logs run in your environment. The agent never touches them directly. It calls MCP servers that do. Your cluster exposes a Kubernetes MCP server, your metrics stack exposes a Prometheus MCP server, and the agent calls both as tools. It never gets a raw kubeconfig or a database connection string.

The first wall anyone hits is that a cloud agent can't reach localhost or a private API server. For local development or a kind cluster, a tunnel like cloudflared's quick-tunnel mode gets you unblocked in minutes. This works for a proof of concept, but not for running production traffic. The production answer is native MCP tunnels. A lightweight gateway inside your network opens a single outbound connection; there's no inbound port and no public MCP endpoint for a scanner to find. This is a general property of cloud-run agents, not something specific to Claude. Any agent platform running outside your perimeter has to solve the same reachability problem, and outbound-only tunneling is the shape the industry, including AWS and Google, has converged on this year.

How do you authenticate an agent to systems it can now reach?

The Managed Agents MCP connector sends one thing to your server: an Authorization: Bearer header. Credentials live in a vault matched by the MCP server's URL, not configured on the server entry itself. This means the hard part of auth isn't the agent side, it's getting your existing auth approach to work with that model.

Three cases show up in practice, in roughly escalating difficulty:

No auth. A Kubernetes MCP server behind a tunnel with cluster-level RBAC as the only gate. Fine for a kind cluster or a low-stakes internal tool; not something to run against a production API server without at least a bearer token in front of it.

Static bearer token. A Grafana service-account token, stored in a vault and matched to the MCP server's URL. This is the common case for read-only observability queries. Prometheus, Loki, and Grafana all support long-lived service-account tokens, and the vault handles rotation.

OAuth client-credentials → JWT. The hard case, since most internal APIs weren't built expecting a bearer-only connector. It sends Authorization: Bearer and nothing else, so the vault has to complete the OAuth exchange up front and hand back a plain bearer token. Sending the client secret as a separate field instead produces a 400 unknown field error, since the connector's schema only expects the one header. For a straightforward approach to handling this see the repo at the end of this post.

The generalizable rule holds regardless of which of the three cases you're in: the connector's interface is Authorization: Bearer. Anything more complex belongs in the vault, not in the agent's configuration.

Once that's wired up and authenticated, the reachability problem is addressed, and the real work can begin.

Access isn't understanding

An agent with full read access to Kubernetes, Prometheus, and your log pipeline still reasons poorly about the system it is investigating. Even the richest telemetry does not come with a model of what depends on what, or more specifically, what can cause what. Metric series, multiple alerts firing, log lines, and even traces provide the agent with rich data to investigate and reason with, but the access to this data is not structured in a way that makes it effective.

Give an agent kubectl, a PromQL endpoint, and a log query tool, and it can retrieve almost anything but retrieval isn't diagnosis. This is not a knock on the underlying telemetry. OpenTelemetry's semantic conventions exist precisely to make spans, metrics, and logs consistent enough to reason over but consistent data isn't the same as a causal model of the system that produced it.

We've written before about why LLMs alone hallucinate root causes for the same underlying reason: pattern-matching over telemetry isn't the same as reasoning over a dependency graph. The distinction between monitoring, knowing something is wrong, and observability, knowing why, makes a related point from a different angle. An agent that can query telemetry has solved the first problem. Solving the second requires encoding dependency and causality structure somewhere: in a semantic layer, a service graph, or an explicit causal model, so the agent has something to reason against, not just something to query.

FAQ

How do I connect a cloud AI agent to infrastructure it can't reach directly? Deploy an MCP server for the system you want the agent to reach, then bridge it with an outbound-only tunnel. For local development or a kind cluster, a quick tunnel like cloudflared works in minutes. For production, use native MCP tunnels: a gateway inside your network opens a single outbound connection, so no inbound port or public MCP endpoint is ever exposed.

Does giving an agent more system access make it more reliable? Not by itself. Raw access to Kubernetes, Prometheus, and logs lets an agent retrieve data, but that doesn't mean it can diagnose. Without an encoded model of service dependencies and event ordering, an agent can't reliably tell a root cause from a downstream symptom. It can query more, not reason better. Access solves reachability; understanding requires a separate structural layer.

What's the difference between an agent reaching a system and an agent understanding it? Reaching a system means the agent can call an MCP server and retrieve metrics, logs, or cluster state. Understanding means the agent can tell which of those signals is the cause and which are downstream effects, something that requires a dependency graph or causal model, not just a wider set of tools to query.

How do I authenticate a managed agent to an internal API that uses OAuth? Resolve the OAuth client-credentials flow inside your credential vault, not in the agent's configuration. The Managed Agents MCP connector only ever sends an Authorization: Bearer header matched to the server's URL. It won't perform a token exchange or accept additional OAuth fields, so the vault has to mint the JWT and return a plain bearer token. See the reference repo below for code examples.

What to do next

Read the three-auth-pattern, kind-runnable reference repo to see all three cases running end to end with a Claude managed SRE agent accessing a real cluster. The open question this setup raises is whether adding a causal or semantic layer on top of raw telemetry access measurably changes how well an agent investigates an incident. We're running a reproducible, measured comparison on this next. Stay tuned.

Semantics in Observability: A Precise Vocabulary for System Understanding

Ben Yemini — Mon, 03 Aug 2026 18:32:30 +0000

Orignially posted to causely.ai by Endre Sara

The word context is overloaded in observability discourse to the point of losing precision. It is used to refer to shared labels, a topology map, temporal proximity, and propagated trace identifiers. Four distinct things, each useful, none of them sufficient.

This blog defines a precise, layered model of observability semantics intended for engineers and SREs building or operating complex distributed systems.

The Gap in Current Observability Tooling

Most observability tooling today implements a shallow semantics of entity and some form of topology, often referred to as the knowledge graph – typically derived from trace data – with partial reach into behavior through threshold-based alerting and anomaly detection. A knowledge graph is a generic term that does not define the semantics of what it captures. It is typically assumed to be a list of services and their topological relationships, but as we point out later, this is not sufficient to answer questions. It is also a dynamic, ever-changing graph; capturing it only once, or once a week, is not a reliable representation for real-time, continuous, automated operation. Deeper semantics are largely absent from tooling and instead exist as undocumented knowledge held by individual engineers.

This lack of semantics has predictable operational consequences:

Alert fatigue: alerting systems fire on symptoms without the ability to identify the root cause, producing storms where a single fault generates dozens of notifications
Slow incident response: root cause reasoning and impact analysis happen manually, in real time, under pressure, because the causal and dependency knowledge exists only in human memory
Blast-radius estimation is guesswork: a lack of knowledge of the impact of failure. There is no systematic way to ask what would happen if a given component failed, which dependent services would be affected, which symptoms would surface, and in what order.
Causal knowledge is reactive, not preventive: no proactive reasoning about causes. Engineers cannot ask what classes of failure could produce a given set of symptoms, or which components in the current system state are most likely to trigger an incident.
No counterfactual reasoning: there is no way to assert that a specific condition could cause a specific failure, or conversely that a given failure mode cannot produce a set of observed symptoms. Without formal causal and dependency knowledge, both directions of reasoning – “could this cause that?” and “could that have caused this?” – rely entirely on individual experience
Knowledge loss: engineers who have internalized the causal and constraint knowledge leave, and the organization reverts to slower, less reliable incident response

The data to address these problems is largely available in service meshes, infrastructure APIs, configuration management systems, and telemetry pipelines. The gap is not data. It is the absence of explicit semantic models that make that data legible to AI Agents. The sections below define the models.

A Layered Semantic Model for Observability

The following six layers describe the semantic structure required to move from “we see a signal” to “we understand what is happening and why.” Each layer builds on the previous. Tooling that skips layers will exhibit predictable gaps in reasoning capability.

It is important to note that the semantics model at each layer has two parts, a knowledge base and a dynamic graph. The knowledge base captures the generic knowledge of what may be observed. The knowledge base captures abstract semantic descriptions common across all environments. The dynamic graph instantiates the knowledge base to generate a real-time model of the managed environment at a given point in time. The dynamic graph is the actual observed reality, specific to a given environment at a given point in time, and keeps adapting as the environment changes.

Layer 1: Entity Model

The entity model defines the named objects in the system and the attributes that describe them. An entity is any discrete thing with identity and an observable state.

The entity knowledge base captures the types of entities that may be discovered, e.g., a service instance, a host, a Kubernetes pod, a database cluster, a message queue, a CDN edge node, an AI Agent, an MCP server with tools or an LLM model and its provider, and the attributes that may be observed:

Identity: UUID with associated attributes like name, namespace, cluster, region, environment, labels
Configuration: resource limits, replica count, connection pool size, JVM flags
Runtime state: CPU utilization, memory used, request rate, error rate, queue depth

The dynamic entity model instantiates the knowledge base with the discovered entities and serves as an inventory of all entities in the environment at a given point in time. For example, the list of services, hosts, pods, databases, message queues, etc., in the environment.

Without an entity model, telemetry is a stream of dimensioned numbers. With one, it is a description of a system composed of named, typed, queryable objects. This semantics is a prerequisite for all higher-order reasoning.

Layer 2: Topology Model

The topology model encodes the structural relationships between entities: which services call which databases, which pods run on which nodes, which queues are consumed by which workers, etc.
The topology knowledge base captures the relationship that may exist between types of entities, For exaple, a service may be connected to another service, a service may be accessing a database, a service may be layered over a pod.

Relationship Types

The dynamic topology graph represents the actual discovered relationships between discovered entities. For example, the Coordinator Agent is connected to the Research, Coding, and Ops subagents; the GitHub MCP server, which includes the create pull request tool; and the GPT-4o model, which is layered over the OpenAI model provider (see Figure 1).

It is important to note that the knowledge base captures design-time assertions, not runtime observations. This distinction is critical. A service dependency map derived from trace data reflects what happened during the observation window. The dynamic topology graph reflects what is. Traces may be incomplete (non-instrumented paths, async operations, batch jobs). The dynamic topology graph enables blast-radius analysis; given that entity foo is degraded, which other entities are structurally dependent on it and therefore at risk.

Figure 1: Topology — structural relationships between agents, MCP servers, and Model Providers.

Layer 3: Behavior Model

A behavior model captures the observable signals of normal and abnormal system behavior. It defines symptoms – semantically meaningful deviations from expected state – and events – discrete state transitions that represent changes in entity health or configuration.

A raw metric is not a symptom. The number 95.3 on a CPU utilization gauge is not meaningful without knowing the entity it describes, the threshold at which that utilization constitutes an anomaly, and the operational context in which that threshold applies. A symptom is the semantic interpretation: this entity is exhibiting CPU saturation relative to its configured capacity and expected workload.

The behavior model is what converts monitoring data into an actionable signal. Alerting systems that operate directly on raw metrics without a behavior model produce high false-positive rates because they lack the semantic layer needed to distinguish signal from noise.

The behavior knowledge base captures the possible observable anomalies an entity may experience. For example, a service may be degraded, experiencing high latency, or a node may be highly utilized, experiencing high CPU utilization.

The dynamic behavior model represents the environment's actual runtime state based on what is being observed. For example, the Research subagent is degraded and experiencing high latency, or the Postgres MCP is congested and experiencing high query times.

Layer 4: Causality Model

The causality model captures the cause-and-effect relationship between potential causes and the symptoms or events they may cause.
The causal knowledge base describes the potential causes, the symptoms they may cause, and how they may propagate. Propagation captures how a cause occurring on one entity may propagate to another related entity.

The causal knowledge base captures generic knowledge about the types of causes that may occur in an environment and how they will manifest, i.e., the symptoms that may be observed when they occur.

For example:

An MCP service may be congested and when it is congested it may cause its tool calls to be slow, manifested as a high tool-call latency symptom, and may propagate and cause degradation of subagents that depend on it.
A subagent may be degraded and when it is degraded it may cause its responses to be slow, manifested as a slow response time symptom, and may propagate and cause degradation of the coordinator agent that orchestrates it.
A model provider may be degraded and when it is degraded it may cause its inference requests to be slow, manifested as high inference latency, and may propagate to subagents using its models causing them to be degraded.

As the above examples illustrate, the causal knowledge base is completely independent of a given environment. It is independent of how many providers, MCP services, and/or subagents are in the environment, and which models run on which providers, which subagents access which MCPs, or which coordinator orchestrates which subagents.

The dynamic causal graph represents the actual runtime causality in the environment at a given point in time. It is a function of the dynamic topology graph and represents all potential specific causes that may occur in the environment, along with the specific symptoms they may cause. The figure below illustrates a concrete example: the Postgres MCP may be congested, and the fault may propagate through the execute_sql tool call to the Coding and Ops subagents, and then onward to the Coordinator, with symptom probability decreasing at each hop. Similarly, Anthropic provider degradation may propagate through Claude inference to the Research, Coding, and Ops subagents, and then to the Coordinator.

In this example, the dynamic causal graph captures that a congested Postgres MCP may cause degradation of the Coding and Ops subagents and the Coordinator. It is important to note that the keyword semantically is “may”. A congested Postgres MCP may degrade downstream entities, but that doesn’t necessarily mean the Postgres MCP is congested – Anthropic provider degradation could produce overlapping symptoms. To conclude, the cause of an observed symptom requires analyzing the state of all potential symptoms.

Figure 2: Causal graph derived from the agentic topology in Figure 1. Two example fault paths (Postgres MCP congestion, Anthropic provider degradation) propagating through MCP services and subagents into the coordinator.

Cause (left) propagates rightward through topology edges. Probabilities indicate the likelihood of each symptom given the cause. Multiple symptoms may share a cause; diagnosis evaluates the full set of observed symptoms against this graph.

The causality model is what makes root cause analysis tractable. Without it, a single infrastructure fault that produces symptoms across dozens of dependent services generates an alert storm with no structural differentiation between cause and effect. With it, symptoms can be suppressed or annotated as consequences once their cause is pinpointed.

Layer 5: Attributes Dependency Model

Where the causality model operates at the symptom level, the attributes dependency model operates at the attribute level. As its name suggests, it captures attribute dependencies. The dependent attributes can be of the same entity or of different entities. Dependency knowledge captures that attributes are correlated, along with the directionality and function of the correlation. The function can be defined in the model or learned.

The attribute knowledge base captures the generic knowledge of which attribute depends on which. Examples of such dependencies:

Subagent response latency is a function of the tool-call duration of the MCP services it invokes
Coordinator response latency is a function of the response latencies of the subagents it orchestrates
Subagent inference latency is a function of the inference latency of the model provider serving its model
A subagent's token throughput is a function of the rate at which the coordinator dispatches requests to it.
The end-to-end coordinator latency is a function of both per-call latencies and the number of subagent calls per task. A non-linear relationship, since a coordinator may fan out to multiple subagents in parallel, retry on failure, or recurse on intermediate results.
Total tokens consumed by a subagent are a function of both the per-call token cost and the number of model-provider inference calls, which scales non-linearly with conversation depth, tool-call retries, and self-reflection loops.

The dynamic attribute dependency graph represents the actual run-time dependencies in the environment at a given point in time. It is a function of the dynamic topology model. Given the above static model examples, the dynamic model examples would be:

If the Coding subagent is accessing the Postgres MCP, Coding subagent latency is a function of execute_sql tool-call duration
If the Coordinator is orchestrating the Coding subagent, Coordinator latency is a function of Coding subagent latency. Note that based on this, Coordinator latency can be computed as a function of execute_sql tool-call duration, even though the Coordinator does not directly access the Postgres MCP.
If the Research subagent is using Claude served by Anthropic, the Research subagent's inference latency is a function of Anthropic's Claude inference latency.
For the Ops subagent, its token throughput is a function of the rate of requests the Coordinator dispatches to it.
If the Coordinator fans out a single user task to the Research, Coding, and Ops subagents in parallel, total Coordinator latency is approximately the max of the three subagent latencies, but if any subagent retries on tool-call failure, that subagent's effective latency multiplies by its retry count, so end-to-end latency grows non-linearly with failure rate.
If the Coding subagent calls the Postgres MCP once per row in a result set, Coding subagent latency is a function of execute_sql duration multiplied by row count, so a query returning N rows produces N tool calls, and end-to-end latency scales with N rather than staying constant.
If the Research subagent invokes Claude in a multi-turn reasoning loop with branching factor B and depth D, total Anthropic inference calls scale as B^D – so doubling reasoning depth roughly squares the inference cost, even though the Research subagent's per-task topology entry is unchanged.

Figure 3: Non-linear call-count amplification. Three runtime patterns where total cost diverges from the static topology in Figure 1. (A) Parallel fan-out with retries multiplies the worst-case branch latency. (B) Per-row tool-call loops produce linear-in-N call counts that the topology graph cannot bound. (C) Recursive reasoning produces B^D inference calls, exponential in depth.

The attribute dependency graph enables change impact analysis: before modifying a configuration parameter, an operator or automation system can trace the dependency graph to identify which runtime attributes will be affected, on which entities, with what expected magnitude.

Layer 6: Constraint Model

The constraint model defines the desired state. The desired state is one in which goals are achieved while constraints are satisfied. Each entity in an environment may have a desired state, but, more interestingly, the environment as a whole may have one as well.

A desired environment state is one in which all applications deliver on their goals while running within their operational constraints. Usually, the goals are performance goals of latency and throughput, and service level goals in terms of errors and availability, while the operational constraints are capacity, budget, compliance, and configuration.

An entity's desired state is the configuration space within which the entity and its attributes operate without triggering symptoms or degradation events. It is the semantic encoding of the system’s operational envelope.

The constraint model captures both the environment's desired state and the entity's desired state. These can be defined independently, but by using the attribute dependency model, the entity's desired state can be generated/inferred from the environment's desired state, resulting in a very compelling and powerful system.

As in all other layers, the constraint knowledge base captures generic knowledge about constraints, and the dynamic constraint model represents the runtime constraints at a given point in time based on the discovered topology. Constraints are not just thresholds on metrics. They are relationships between configuration parameters and the runtime conditions required for healthy operation:

Headroom constraint: Container memory limit >= 1.4 x used heap
Budget constraint: Application cost < budget
Latency constraint: Service latency < 100ms

When the constraint model is explicit, it becomes possible to answer operational questions that today require senior engineering intuition: “What is the minimum safe memory limit for this service at current traffic?” “If we reduce the connection pool by 40%, at what request rate will we begin to see queuing symptoms?” The constraint model is the missing link between observability (what is happening) and operations (what configuration produces health). It is the semantic layer connecting configuration management to runtime behavior.

What “Context” Actually Covers and Where It Falls Short

Most observability platforms use the word loosely. Here is what the four common implementations actually encode, and where each one stops.

Shared Labels
Common attributes – service, environment, region, version – are attached to metrics, logs, and traces. This allows tooling to pivot between signal types for a given entity. The relationship is implied by label equality, not by any explicit model of dependency or causation. Two data points sharing service=checkout are navigable together; they are not necessarily related by anything other than that label. This is a partial implementation of the entity model (layer 1) that lacks declarative topological relationships and behavior.

Topology Map
A topology map encodes structural relationships between entities, which services call which databases, which pods run on which nodes, and which workloads share a cluster. It is the most semantically substantive form of “context” in common tooling and is typically derived from trace data, service-mesh telemetry, or infrastructure APIs. This is a partial implementation of the topology model (layer 2): it captures that entity X is connected to entity Y, but not what behavior propagates along that edge, why a fault on X causes a symptom on Y, or which of X’s attributes drives which of Y’s. A topology map shows the wiring; it does not encode behavior (layer 3), causality (layer 4), attribute dependency (layer 5), or constraints (layer 6). Two services connected in the topology may be tightly coupled in failure propagation or entirely decoupled. The map alone cannot distinguish them. In practice, this means a topology map supports navigation (“show me what this service talks to”) but not reasoning (“if this service degrades, what symptoms should I expect, on which dependents, and why”). That reasoning requires the higher layers.

Temporal Proximity
Events co-located on a timeline are assumed by operators to be potentially related. The system carries no model of that relationship; pattern recognition is entirely delegated to the human. This is correlation by coincidence, not by structure. It addresses none of the six semantic layers explicitly; it is a UI affordance for human reasoning, not a semantic model.

Distributed Trace Propagation
A trace context header (W3C traceparent, B3, etc.) propagated across service boundaries encodes a genuine causal chain: span B was initiated by span A. This is the most semantically rich form of “context” in common use. It partially implements the topology model (layer 2) for instrumented call paths, and provides raw material for the behavior model (layer 3). However, it is constrained to synchronous, instrumented code paths and does not capture async operations, batch jobs, or infrastructure-level dependencies. It carries no causality, dependency, or constraint models.

In summary, shared labels, temporal proximity, topology, and trace propagation collectively reach into layers 1 and 2. Layers 4 through 6 – causality, dependency, and constraints – are entirely outside the scope of what “context” addresses. That is the gap.

Conclusion

Context – shared labels, topology map, temporal proximity, trace propagation – is a necessary foundation for navigating observability data. It is not sufficient for understanding systems. The jump from “we can see the signals” to “we understand what is happening and why” requires explicit semantic models at six distinct layers: entity, topology, behavior, causality, dependency, and constraint.

The practical implication for SREs and platform engineers is that runbooks, tribal knowledge, and escalation paths are informal implementations of layers 4 through 6. The question is not whether to encode this knowledge, which already exists, but whether to leave it implicit in human memory or make it explicit in a form that tooling can reason about.
Making semantics explicit is what makes a system legible to an AI agent. An agent reasoning over shared labels, topology, and temporal proximity alone is restricted to the same navigation problem a human faces, pivoting between signals without a model of why they are related. An agent reasoning over layers 4 through 6 can answer the questions that actually drive incident response, capacity planning, and change approval: what is causing this symptom, what is the blast radius of a degraded component, what configuration change would restore the constraint, and what would happen if that change were made. These questions do not require new data; the telemetry, configuration, and trace propagation already carry the raw signal. What they require is a model that the agent can reason against. One that encodes the causal, dependency, and constraint knowledge that today lives only in the engineers who happen to remember.

Semantics is not an observability feature. It is the foundation that makes observability useful.