<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ben Yemini</title>
    <description>The latest articles on DEV Community by Ben Yemini (@byemini13).</description>
    <link>https://dev.to/byemini13</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3427886%2Fbfc569e8-b092-4be7-abd9-72bc4ae1d20c.jpeg</url>
      <title>DEV Community: Ben Yemini</title>
      <link>https://dev.to/byemini13</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/byemini13"/>
    <language>en</language>
    <item>
      <title>Your agents trust your telemetry. Should they?</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 16 Nov 2026 20:37:00 +0000</pubDate>
      <link>https://dev.to/causely/your-agents-trust-your-telemetry-should-they-4ie0</link>
      <guid>https://dev.to/causely/your-agents-trust-your-telemetry-should-they-4ie0</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/your-agents-trust-your-telemetry-should-they" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/jpkroehling/" rel="noopener noreferrer"&gt;Juraci Paixão Kröhling&lt;/a&gt; and &lt;a href="https://www.linkedin.com/in/byemini/" rel="noopener noreferrer"&gt;Ben Yemini&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A joint post by OllyGarden and Causely.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Zero errors, healthy service
&lt;/h2&gt;

&lt;p&gt;An agent is asked whether a payment service had problems in the last hour. It queries the logs for records at ERROR severity, gets an empty result, and reports that the service is healthy. The query was correct. The logs were not. The service wrote lines like &lt;code&gt;level=ERROR msg="executor failed" err="unexpected EOF"&lt;/code&gt;, but the level only lived in the text of the message. The severity field on every one of those records was empty, so the filter matched nothing.&lt;/p&gt;

&lt;p&gt;A person scrolling through the raw logs would probably have spotted the word ERROR. An agent that does what it is told, and does it fast, has no reason to look. It reads an empty result as an answer, and in this case the answer was wrong.&lt;/p&gt;

&lt;p&gt;That is the first of two questions this post sets out to answer: can your agents trust the telemetry they read? The second is harder, and it takes up the second half: even when the data is trustworthy, can an agent tell what it means?&lt;/p&gt;

&lt;h2&gt;
  
  
  Part one: Can the data be trusted?
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;By OllyGarden&lt;/em&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Bad telemetry versus untrustworthy telemetry
&lt;/h3&gt;

&lt;p&gt;Some bad telemetry announces itself. A log storm that writes the same line ten thousand times a minute is obviously wrong, and so is a password sitting in a log attribute. Anyone who looks at that data, human or agent, knows something needs fixing. Those are real problems, with real costs and real risks, but they are visible.&lt;/p&gt;

&lt;p&gt;Untrustworthy telemetry is a different kind of problem. It is well-formed, it fits the schema, and it returns plausible results. It just does not say what happened. An agent queries it and gets an empty result, a smaller count, or a healthy-looking service, and nothing in the response suggests the answer is wrong. That is the telemetry this post is about.&lt;/p&gt;

&lt;p&gt;We audit OpenTelemetry data for a living. Between July and October 2026 we sampled telemetry from multiple organizations and looked for the problems that change what a query returns. The sample size isn’t representative of all organizations out there, and all of them are representing teams that chose to work with us on telemetry quality, so treat the numbers below as what we see in practice, not as an industry survey. We group what we found into three ways telemetry loses trust.&lt;/p&gt;

&lt;h3&gt;
  
  
  Silent absence: the data is not there
&lt;/h3&gt;

&lt;p&gt;The most common problem is data that should exist and does not. An agent cannot reason about what it cannot see, and nothing in the result tells it that something is missing.&lt;/p&gt;

&lt;p&gt;Error spans are the clearest case. In every organization where we checked trace data in depth, between 38% and 96% of spans with an error status had no status message and no exception event, and most were above 90%. The agent learns that something failed but not why, and has to guess the cause from the span name and duration.&lt;/p&gt;

&lt;p&gt;Ownership goes missing too. Nearly every organization sent telemetry without a usable &lt;code&gt;service.name&lt;/code&gt;, either with no value at all or with the SDK default, such as &lt;code&gt;unknown_service:java&lt;/code&gt;. For an agent, that telemetry belongs to nobody. It cannot route the problem to a team, correlate it with a deployment, or compare it against the service's normal behavior.&lt;/p&gt;

&lt;p&gt;Then there are traces that stop at the first hop. Every organization in the sample had services where most traces consisted of a single span, ranging from 59% to 100% of their traces, with one service at 99.9% of 1.4 million traces in a window of 5 minutes. Some single-span traces are legitimate, such as a worker processing one job. At these rates, though, they usually point to context propagation breaking at a gateway, a queue, or a thread pool. An agent following a slow request across services reaches that boundary and stops, and the hop it ends up blaming is the last one it can see, not the one at fault.&lt;/p&gt;

&lt;h3&gt;
  
  
  Contradiction: the fields disagree
&lt;/h3&gt;

&lt;p&gt;The second way is telemetry that describes the same event in two incompatible ways. A human weighs both and gets suspicious. An agent usually trusts whichever field its query touches.&lt;/p&gt;

&lt;p&gt;Log severity is where we see this most. In most organizations we checked, between roughly a quarter and three quarters of log records whose body reads like an error were recorded at INFO, at DEBUG, or with no severity at all. One record we found carried its own contradiction: {"error":"context canceled","level":"INFO","msg":"Unable to proxy request"}. Another service logged &lt;code&gt;Error starting ApplicationContext&lt;/code&gt; at INFO during a failed startup.&lt;/p&gt;

&lt;p&gt;Spans do this too. One span we saw reported HTTP status 200 for a batch export, while the gRPC status underneath was 8, resource exhausted, with a message saying the batch exceeded the maximum message size. An agent checking the HTTP status concludes that the batch was delivered. It was rejected.&lt;/p&gt;

&lt;h3&gt;
  
  
  Split vocabulary: the query sees part of the data
&lt;/h3&gt;

&lt;p&gt;The third way is the hardest to spot, because every query looks correct. The telemetry uses more than one name for the same thing, and the query only knows one of them.&lt;/p&gt;

&lt;p&gt;Every organization we checked had more than one generation of attribute names in flight at the same time. In one of them, 125 services reported &lt;code&gt;http.method&lt;/code&gt; while 16 reported http.request.method. An agent that writes its query against the current conventions sees about an eighth of the services and draws conclusions about traffic and error rates from that slice. Severity has the same problem: one organization used 30 different spellings for its log levels, including &lt;code&gt;ERR&lt;/code&gt;, &lt;code&gt;error&lt;/code&gt;, &lt;code&gt;SEVERE&lt;/code&gt;, and a JSON fragment that leaked into the field. Most organizations still used &lt;code&gt;deployment.environment&lt;/code&gt; alongside, or instead of, &lt;code&gt;deployment.environment.name&lt;/code&gt;, so a filter for production returns some of production.&lt;/p&gt;

&lt;h3&gt;
  
  
  The fix is a deliberate minimum
&lt;/h3&gt;

&lt;p&gt;None of these problems are solved by collecting more data. More auto-instrumentation adds more spans with the same missing reasons, more logs with the same missing severity, and more volume for an agent to read. What helps is a small set of telemetry that is correct on purpose.&lt;/p&gt;

&lt;p&gt;We call this Minimum Viable Instrumentation (MVI). It instruments the boundaries of each service: inbound requests, outbound calls to databases, queues, and other services, and the context propagation that ties them together. It records errors with a reason. It settles on one set of attribute names across the organization. And it is verified against a real Collector before it is merged, so a broken &lt;code&gt;service.name&lt;/code&gt; or a lost trace context is caught in review, not during an incident. We wrote about the approach in &lt;a href="https://ollygarden.com/resources/blog/introducing-minimum-viable-instrumentation?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Introducing Minimum Viable Instrumentation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;When to skip this: if a service is about to be retired, or nobody, human or agent, will ever query its telemetry, fixing its instrumentation is not a good use of time. Start with the services your agents already read. The Pareto principle applies here as well: prioritize the top 20% of critical services to achieve the maximum impact.&lt;/p&gt;

&lt;h3&gt;
  
  
  When good data still misleads
&lt;/h3&gt;

&lt;p&gt;Making telemetry trustworthy removes one class of wrong answers, but not all of them. Some telemetry is well-formed, consistent, and correctly instrumented, and still easy to misread.&lt;/p&gt;

&lt;p&gt;A refund worker reports a p99 latency of 20 seconds, which looks like a severe slowdown until you notice that 20 seconds is the queue's polling interval, so the "latency" measures how long the worker waited for work. A trace that appears to start at a ledger consumer has a root span that still references a parent, because the producer that created it never exported its spans.&lt;/p&gt;

&lt;p&gt;Neither of these is untrustworthy data. The values are accurate, and no quality check would flag them. Reading them correctly requires knowing how the system fits together — and that is where Causely picks up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part two: Can an agent tell what the data means?
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;By Causely&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Those two cases are the shape of the second problem. The p99 that measures waiting and the trace whose producer never reported are both correct, and both invite a confident wrong answer. Fix every field in the logs from the opening, and an agent can still miss the payment service entirely, even when every signal it reads is accurate.&lt;/p&gt;

&lt;h3&gt;
  
  
  An agent opens a PR
&lt;/h3&gt;

&lt;p&gt;Say you run a background agent that watches for reliability issues and opens a pull request when it finds one. Here is how that can go wrong, using the checkout flow in the OpenTelemetry Astronomy Shop demo app, where checkout calls payment to charge each order, and an illustrative fault.&lt;/p&gt;

&lt;p&gt;Checkout's error alert fires. The agent picks it up, pulls checkout's recent errors, logs, and traces, and makes two observations: requests are failing, and checkout's calls to product-catalog hit connection-refused errors from a product-catalog pod that restarted earlier that afternoon. Every trace is intact, and both observations are real.&lt;/p&gt;

&lt;p&gt;The agent comes to a reasonable conclusion: checkout does not handle a flaky dependency well. It opens a PR that adds retries with backoff to checkout's product-catalog client, with a clear description and evidence linking the restart to the errors. It is the kind of PR a busy reviewer approves.&lt;/p&gt;

&lt;p&gt;The actual problem is a code change in payment that rejects every transaction. Payment never alerted, and there were no restarts, OOMKills, or resource problems on its pods. The errors in checkout, accounting, and shipping all trace back to it.&lt;/p&gt;

&lt;p&gt;After the merge, checkout is still failing. The team has spent a review cycle on a change nobody needed, and because a fix shipped, the problem can look handled while transactions keep failing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why the agent got it wrong
&lt;/h3&gt;

&lt;p&gt;The agent did not make anything up. Every piece of evidence it used was in the telemetry, and its reasoning from that evidence was sensible.&lt;/p&gt;

&lt;p&gt;What it lacked was a way to separate cause from effect. Starting from the alert, it searched outward until something explained the errors, and a restart on a dependency explained them well enough. Payment raised no alert and showed no infrastructure symptoms, so nothing pointed the agent there first. A better model or a longer prompt does not fix this, because the misleading evidence is real.&lt;/p&gt;

&lt;p&gt;It is also the expensive path. Every extra log query or probe costs tokens and time, and gives the agent another chance to find something that fits. In many cases the gap is wider than one application. An infrastructure issue can surface as application errors, and a slow external dependency can set off alerts in components that did nothing wrong.&lt;/p&gt;

&lt;h3&gt;
  
  
  The same alert, with causal context
&lt;/h3&gt;

&lt;p&gt;Causely is a causal reasoning system that continuously maintains a live map of your environment: what depends on what, how each kind of fault propagates to the parts of the stack around it, and which active symptoms point back to which cause (more on the &lt;a href="https://www.causely.ai/blog/semantics-in-observability?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;semantics behind this here&lt;/a&gt;). Through Causely's MCP server, the agent can check that model before it starts searching.&lt;/p&gt;

&lt;p&gt;Here, it can see that the errors in checkout, accounting, and shipping point back to payment, and that the product-catalog restart is unrelated and already over. The PR it opens targets the payment change, and the description says why: which services were affected, which cause explains them, and what it ruled out. The reviewer checks an argument, including the causal graph, rather than redoing the investigation. The agent doesn't have to wait for checkout's alert at all. Causely can trigger it directly, with payment already identified as the cause, so the investigation starts from the answer.&lt;/p&gt;

&lt;h3&gt;
  
  
  The numbers behind it
&lt;/h3&gt;

&lt;p&gt;We ran this fault in a public benchmark on the &lt;a href="https://opentelemetry.io/docs/demo/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;OpenTelemetry Astronomy Shop&lt;/a&gt;, with four agent configurations — two coding agents and two SRE agents — each with and without Causely's MCP server. The samples are small, so read the size of the gap rather than the exact values:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Without causal context&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;With Causely&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cause questions answered correctly&lt;/td&gt;
&lt;td&gt;75%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool calls per investigation&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens per run&lt;/td&gt;
&lt;td&gt;308K&lt;/td&gt;
&lt;td&gt;123K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time to diagnosis&lt;/td&gt;
&lt;td&gt;73 seconds&lt;/td&gt;
&lt;td&gt;25 seconds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;False incidents reported on a healthy cluster&lt;/td&gt;
&lt;td&gt;4 of 12 runs&lt;/td&gt;
&lt;td&gt;1 of 12 runs&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;We saw the same pattern in a background-agent setup. With &lt;a href="https://www.causely.ai/blog/how-to-make-claude-background-agent-3-5x-faster-and-cheaper?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Claude Managed Agents on a 36-service app&lt;/a&gt;, both agents found the right fix, but the one without causal context made 3.6x to 7x more tool calls. Full results are on the &lt;a href="https://www.causely.ai/product/benchmark?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;benchmark page&lt;/a&gt; and in the &lt;a href="https://arxiv.org/abs/2605.18327?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;paper&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the two halves meet
&lt;/h2&gt;

&lt;p&gt;Causely's reasoning draws on your telemetry, so the problems in the first half affect it too. A service without a usable &lt;code&gt;service.name&lt;/code&gt; can end up with no clear identity in the dependency graph, and broken context propagation can hide connections between services. Causely closes some of these gaps itself: its eBPF auto-instrumentation fills in missing coverage, and because it reasons from how faults propagate, it can identify causes that were never directly instrumented. When inputs are poor, the diagnosis gets less specific rather than wrong. Clean telemetry makes that diagnosis sharper, and causal reasoning is what lets an agent use clean telemetry well.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist: can your agents trust what they read?
&lt;/h2&gt;

&lt;p&gt;Nine things to check. The first five are queries you can run against your telemetry this week. The last four are questions to ask of your agent's recent output.&lt;/p&gt;

&lt;h3&gt;
  
  
  On the data from OllyGarden
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;What share of your error spans carry a reason, either a status message or an exception event?&lt;/li&gt;
&lt;li&gt;Which services report &lt;code&gt;unknown_service&lt;/code&gt; or no &lt;code&gt;service.name&lt;/code&gt; at all?&lt;/li&gt;
&lt;li&gt;What share of traces are a single span, broken down by service?&lt;/li&gt;
&lt;li&gt;How many attribute names are in use for HTTP methods, log severity, and deployment environment?&lt;/li&gt;
&lt;li&gt;Do log severity and log body agree? Search error-like bodies and count how many are not at ERROR or WARN.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  On the reasoning from Causely
&lt;/h3&gt;

&lt;p&gt;6. Of your agent's last ten reliability PRs, how many changed the service that alerted rather than the one that caused the alert?&lt;br&gt;
7. When you ask your agent what it ruled out, can it name alternatives and say why it rejected them?&lt;br&gt;
8. Does the evidence in each PR line up in time with the problem it claims to fix? A restart that ended before the errors started does not explain them.&lt;br&gt;
9. After your agent's fixes merge, how often does the original alert fail to clear?&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;Every telemetry example in the first half of this post comes from real data, and we turned twenty of them into a short game: each round shows an anonymized snippet and a claim, and you decide whether to trust it. Play it at &lt;a href="https://ollygarden.com/trust-your-telemetry?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;ollygarden.com/trust-your-telemetry&lt;/a&gt;. If you get fooled, your agents probably would be too.&lt;/p&gt;

&lt;p&gt;The benchmark queries, ground-truth answers, and setup from the second half are in &lt;a href="https://github.com/causely-oss/otel-demo/blob/main/BENCHMARK.md?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Causely's repo&lt;/a&gt;, with full results on the &lt;a href="https://www.causely.ai/product/benchmark?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;benchmark page&lt;/a&gt; and on &lt;a href="https://arxiv.org/abs/2605.18327?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;arXiv&lt;/a&gt;. If you run it and get different results, open an issue.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzmqdhydgfn6osi9bdysf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzmqdhydgfn6osi9bdysf.png" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>observability</category>
      <category>opentelemetry</category>
      <category>ai</category>
      <category>devops</category>
    </item>
    <item>
      <title>What Is a Causal Model in Observability?</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 09 Nov 2026 17:45:27 +0000</pubDate>
      <link>https://dev.to/causely/what-is-a-causal-model-in-observability-22b</link>
      <guid>https://dev.to/causely/what-is-a-causal-model-in-observability-22b</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/what-is-a-causal-model-in-observability" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/yyemini/" rel="noopener noreferrer"&gt;Yotam Yemini&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In 2026, the word "model" seems to mean one thing to almost everyone: a large language model. So when we tell customers that Causely is a system that builds and maintains live causal models of their environment, a lot of people interpret that to mean we're training or tuning an LLM on their data.&lt;/p&gt;

&lt;p&gt;To be clear, we are not.&lt;/p&gt;

&lt;p&gt;The causal models built and maintained by Causely contain no neural network, don't produce answers by predicting the next word, and are never trained on your data. Here's what they are and how they work.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Causal Model Is Knowledge of Cause and Effect
&lt;/h2&gt;

&lt;p&gt;A causal model describes how the behavior of one kind of component affects the components around it. Consider an exhausted database connection pool as an example. It raises latency for anything querying that database, which can cause timeouts in the services calling those, which can surface as errors at the edge.&lt;/p&gt;

&lt;p&gt;The models are generic: they encode how effects propagate in general, not how your particular environment is wired. That database chain holds whether the database runs in Kubernetes, on a VM, or as a managed cloud service.&lt;/p&gt;

&lt;p&gt;You can also inspect the models and extend them with causes specific to your stack. Nothing about how Causely reaches an answer is hidden. &lt;/p&gt;

&lt;h2&gt;
  
  
  How General Knowledge Becomes Specific to Your Environment
&lt;/h2&gt;

&lt;p&gt;Causely works from a live map of your environment: every component across infrastructure, platform, and application layers, plus your third-party dependencies, and how they all connect. Causely can discover and maintain that map itself, read one your team already maintains (such as a topology in your data lake), or combine the two and fill in the gaps.&lt;/p&gt;

&lt;p&gt;It then applies the general causal knowledge to that map. The result covers your specific environment at that moment: typically tens of thousands of possible causes and millions of cause-symptom combinations. Those results can be queried by your agents over MCP, automatically trigger a background agent, or be written into your own knowledge graph or data platform as often as you need.&lt;/p&gt;

&lt;p&gt;When symptoms appear, Causely works out which cause best explains them. That inference is deterministic: the same symptoms produce the same answer every time. It can also pinpoint a cause you have no direct visibility into, such as an external dependency, because that cause still shows up in the symptoms it creates in the parts of your stack around it.&lt;/p&gt;

&lt;p&gt;If the evidence can’t pinpoint a specific cause, Causely reports what it can conclude rather than guessing. For example, it may identify that a particular component is degraded and is causing the anomalies on everything that depends on it, without concluding why it’s degraded if the evidence doesn’t show that.&lt;/p&gt;

&lt;p&gt;As your environment changes, the results update with it. There is no need to write conditional rules or runbooks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Causal Models vs. Rules Engines
&lt;/h2&gt;

&lt;p&gt;A rules engine encodes specific scenarios: if these alerts fire together, the cause is X. Someone must write each rule for each environment and rewrite it every time the environment changes. A scenario nobody wrote a rule for goes unexplained.&lt;/p&gt;

&lt;p&gt;A causal model encodes how effects propagate between types of components, independent of any one environment. Causely applies it to your live map and derives the specific scenarios automatically, including ones nobody anticipated. When your environment changes, the scenarios change with it.&lt;/p&gt;

&lt;p&gt;Rules also struggle when several things go wrong at once, because overlapping symptoms can match multiple rules or none. A causal model weighs all the observed symptoms together and identifies the cause, or causes, that best explain them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Causal Models vs. Correlation-Based Analysis
&lt;/h2&gt;

&lt;p&gt;Plenty of observability platforms claim to provide some form of "causal AI" or "causal analysis." In many cases, what sits underneath is a correlation engine built to find the signals that turned anomalous around the same time, filter them by topology, and rank the likeliest culprit.&lt;/p&gt;

&lt;p&gt;Correlation tells you what changed together, but it doesn't tell you whether A caused B, B caused A, or both were caused by C. When several parts of your stack degrade at once, the earliest or most visible anomaly often gets blamed, even when it's a symptom of something else. A common example is an API gateway: every request passes through it, so its errors and latency line up with nearly every issue, and it gets blamed even when the cause sits behind it.&lt;/p&gt;

&lt;p&gt;One other major limitation of correlation-based approaches is that it only works with signals it can see. A cause with no direct telemetry, such as an issue inside a third-party dependency, can't be flagged because it isn't in the data. A causal model can still identify it from the symptoms it produces around it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Causely Matters for SRE and DevOps Agent Builders
&lt;/h2&gt;

&lt;p&gt;On their own, agents are good at searching and summarizing data and drawing plausible conclusions from it, but plausible isn't proven. When several explanations fit the data, they can't reliably tell which one is the actual cause, so they pattern-match until one looks right. That applies whether the agent is triaging a live issue, assessing the risk of a change, or looking for emerging problems.&lt;/p&gt;

&lt;p&gt;With Causely, an agent queries cause and effect directly instead of reconstructing it every time. In our &lt;a href="https://www.causely.ai/product/benchmark?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;benchmark&lt;/a&gt;, agents with Causely diagnosed 63% faster at 57% lower cost, and hallucinated incidents dropped to zero.&lt;/p&gt;

&lt;p&gt;The causal knowledge Causely maintains also helps prevent issues, in two ways. First, Causely infers the causes your environment could produce before any of them occur, so you can see what a planned change puts at risk before it ships. Second, a background agent connected to Causely can catch emerging issues from their early symptoms and address them before they affect users.&lt;/p&gt;

&lt;p&gt;In short, a causal model is explicit, inspectable knowledge of cause and effect across your environment. Your agents and engineers can query it directly instead of spending time and tokens building and testing hypotheses about what caused what, otherwise known as guessing.&lt;/p&gt;

&lt;h2&gt;
  
  
  See It In Your Own Stack
&lt;/h2&gt;

&lt;p&gt;Getting started with Causely is easy. Check out how in our &lt;a href="https://docs.causely.ai/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;docs&lt;/a&gt; or &lt;a href="https://auth.causely.app/oauth/account/sign-up?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;sign up&lt;/a&gt; to start free, no credit card required.&lt;/p&gt;

</description>
      <category>observability</category>
      <category>ai</category>
      <category>devops</category>
      <category>sre</category>
    </item>
    <item>
      <title>Causely Feature Demo: Diagnosing Failures in an Agentic App</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 02 Nov 2026 16:08:47 +0000</pubDate>
      <link>https://dev.to/causely/causely-feature-demo-diagnosing-failures-in-an-agentic-app-a9k</link>
      <guid>https://dev.to/causely/causely-feature-demo-diagnosing-failures-in-an-agentic-app-a9k</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/causely-feature-demo-agentic-app" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/anson-mccook-60a22944/" rel="noopener noreferrer"&gt;Anson McCook&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/bBsdISbhQiY" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;An agentic app adds new places for a failure to start. The agent calls tools on an &lt;a href="https://modelcontextprotocol.io/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;MCP server&lt;/a&gt;, each tool calls services, and those services call databases. When a run fails, the agent reports a schema error or a failed session. The cause can sit in a tool's response contract, in the MCP server's code, in a service, or in the infrastructure. Each layer shows a different symptom, and none of them names the cause.&lt;/p&gt;

&lt;p&gt;Causely maps the whole app automatically: infrastructure, services, the request flow between them, and the agent's activity. Every agent session is linked to the tools it called, and every tool to the service behind it. Causely builds a causal model on top of that topology and reasons from the symptoms to the root cause.&lt;/p&gt;

&lt;p&gt;We ran this on &lt;a href="https://github.com/instana/robot-shop?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Robot Shop&lt;/a&gt; with an agentic shopping assistant. Each run searches the catalog, compares two robots, adds one to the cart and verifies the cart, through a real agent framework, an MCP server, and the services and databases behind it. Then we injected two failures. Both look the same from the agent's side. They start in different layers.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Fault injected&lt;/th&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What the agent saw&lt;/th&gt;
&lt;th&gt;Causely's diagnosis&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;search_products&lt;/code&gt; returns price as text&lt;/td&gt;
&lt;td&gt;Tool contract&lt;/td&gt;
&lt;td&gt;Every run fails (schema error)&lt;/td&gt;
&lt;td&gt;MCP tool contract violation on &lt;code&gt;search_products&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unhandled Python exception on every request&lt;/td&gt;
&lt;td&gt;MCP server code&lt;/td&gt;
&lt;td&gt;Every run fails&lt;/td&gt;
&lt;td&gt;Unhandled Python exception in &lt;code&gt;shop-mcp&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The contract failure is the harder one to spot, because the MCP server still reports success. Causely reached each diagnosis within a few minutes. For the tool fault, the evidence clears the agent and the infrastructure. For the server fault, it ties the exception in the server's logs to error signals on &lt;code&gt;shop-mcp&lt;/code&gt; and failed sessions on &lt;code&gt;shopping-load&lt;/code&gt;. The causal view traces each fault to every agent session and request path it breaks, with a probability on each link in the causal chain.&lt;/p&gt;

&lt;p&gt;In production, thousands of agents and apps call one MCP server that exposes many tools, with services, databases and infrastructure behind them. A failure can start in any of those layers. Causely connects the symptoms back to the one cause behind them.&lt;/p&gt;

&lt;p&gt;Watch the video to see both failures diagnosed in Robot Shop. To see how Causely covers AI workloads, read &lt;a href="https://www.causely.ai/product/agentic-apps?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Observe AI Workloads&lt;/a&gt;. For results across agents and scenarios, see the &lt;a href="https://www.causely.ai/product/benchmark?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Causely benchmark&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>mcp</category>
      <category>observability</category>
      <category>devops</category>
    </item>
    <item>
      <title>Causely Upgrades Its MCP Server So Background Agents Are Faster to Deploy, Easier to Trust, and Cheaper to Run</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 26 Oct 2026 14:46:41 +0000</pubDate>
      <link>https://dev.to/causely/causely-upgrades-its-mcp-server-so-background-agents-are-faster-to-deploy-easier-to-trust-and-2glf</link>
      <guid>https://dev.to/causely/causely-upgrades-its-mcp-server-so-background-agents-are-faster-to-deploy-easier-to-trust-and-2glf</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/causely-mcp-server-upgrades-background-agents" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/company/causely-ai/" rel="noopener noreferrer"&gt;Causely&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Updates shaped by production use at Unity and Quantum Metric; now listed in the GitHub MCP Registry with new open source examples&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;NEW YORK, October 22, 2026.&lt;/strong&gt; Causely, the causal intelligence layer for AI agents operating production systems, today announced upgrades to its MCP server based on how background agents use it in production. The release makes agents faster to deploy, easier to trust, and cheaper to run, and reflects months of learning from how agents use it at customers including Unity and Quantum Metric, and in Causely's own environment. Causely's MCP server is now listed in the GitHub MCP Registry, and new working examples of background agents are available in Causely's open source repositories. The announcement coincides with AGNTCon + MCPCon North America 2026, where Causely is a sponsor.&lt;/p&gt;

&lt;p&gt;Background agents are taking on a growing share of the work that keeps applications performant. They run on a schedule or a trigger, investigate on their own, and hand back a diagnosis or a fix. Teams building them keep hitting the same three problems: the agent takes weeks of prompt engineering before it uses tools well, when it acts, nobody can easily check what it based the decision on, and it burns tokens rediscovering the state of the system on every run.&lt;/p&gt;

&lt;p&gt;This release addresses each of those.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Faster to deploy.&lt;/strong&gt; Causely's MCP server now carries the guidance an agent needs to use its tools well, so a new agent is productive on first connection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Easier to trust.&lt;/strong&gt; Every agent action can be traced back to the diagnosis and evidence it was based on, and teams can see how their agents are using Causely over time. A pull request or Slack message from an agent now comes with a trail a human can check.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cheaper to run.&lt;/strong&gt; Agents can start from what is wrong and what recently changed, delivered as structured results, rather than scanning the environment to work it out. Causely's published benchmark across 72 experiments showed agents using Causely reached 100% fault detection accuracy while using 48% fewer tokens and returning answers 63% faster.&lt;/p&gt;

&lt;p&gt;Unity runs background agents on Causely's MCP server across thousands of services. Multiple teams have integrated Causely into their agents, including an internally developed AI SRE and agents built on Claude, HolmesGPT, and Hermes. Causely is the first tool those agents call when investigating an incident, and the same agents run continuous scans of the environment to identify reliability blind spots. Quantum Metric connects Causely to the agent its engineers work with while investigating problems, so the first answer an engineer gets back already includes the cause and what it affects.&lt;/p&gt;

&lt;p&gt;"Our engineers work across thousands of services. With Causely behind the agent they use, they get a better answer, faster, and one they can act on with confidence," said Glenn Trattner, COO of Quantum Metric.&lt;/p&gt;

&lt;p&gt;"We have spent the past year seeing how background agents run against Causely, at customers and in our own environment, and the lesson was consistent. An agent is only as good as its understanding of the system it operates on. This release puts more of that understanding into the MCP server itself, so agents are productive sooner, teams can trust what they act on, and every run costs less," said Yotam Yemini, CEO of Causely.&lt;/p&gt;

&lt;p&gt;The updates are available now to all Causely customers at &lt;code&gt;api.causely.app/mcp&lt;/code&gt; and work with any MCP-compatible client, including Claude Code, Cursor, Codex, Hermes, and HolmesGPT. The server is also listed in the GitHub MCP Registry at &lt;a href="https://github.com/mcp/app.causely/causely?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;github.com/mcp/app.causely/causely&lt;/a&gt;. Background agent examples and Causely MCP Skills are open source at &lt;a href="https://github.com/causely-oss?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;github.com/causely-oss&lt;/a&gt;. Causely will be at booth 80 at AGNTCon + MCPCon North America, October 22 to 23.&lt;/p&gt;

&lt;h2&gt;
  
  
  About Causely
&lt;/h2&gt;

&lt;p&gt;Causely provides the causal intelligence layer for AI agents operating production systems. Its continuously maintained causal model turns telemetry into structured, machine-consumable understanding of cause and effect, delivered through MCP, so agents diagnose accurately, use fewer tokens, and act sooner. Causely is a Silver member of the Linux Foundation, CNCF, and the Agentic AI Foundation. Learn more at &lt;a href="https://www.causely.ai/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>mcp</category>
      <category>devops</category>
      <category>observability</category>
    </item>
    <item>
      <title>How OpenTelemetry observes AI agents without code changes</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 19 Oct 2026 22:42:16 +0000</pubDate>
      <link>https://dev.to/causely/how-opentelemetry-observes-ai-agents-without-code-changes-bm3</link>
      <guid>https://dev.to/causely/how-opentelemetry-observes-ai-agents-without-code-changes-bm3</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/opentelemetry-ai-agent-observability-obi" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/endresara" rel="noopener noreferrer"&gt;Endre Sara&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;AI agents are hard to operate because instrumentation can't keep up with them. Haibin Zhang of Alibaba Cloud and I made that case at KubeCon China 2026, and showed how OpenTelemetry eBPF Instrumentation (OBI) closes the gap. The full talk is at the end of this post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Multi-agent applications fail in ways request metrics miss
&lt;/h2&gt;

&lt;p&gt;A traditional request follows a static path: client, gateway, service, database. An agent request is dynamic. One prompt fans out into embeddings, vector search, reranking, tool calls, and other agents, and the path changes on every run.&lt;/p&gt;

&lt;p&gt;Haibin shared an example. One customer's token usage on AWS Bedrock and OpenAI grew 320% in a month, and the team could not tell which agent drove it. Other teams call LLM APIs over raw HTTP without a provider SDK, so SDK instrumentation sees nothing.&lt;/p&gt;

&lt;p&gt;Library coverage is also fragmented. In Python, Bedrock instrumentation works but Anthropic may not. MCP instrumentation exists for npm but not for Go.&lt;/p&gt;

&lt;h2&gt;
  
  
  OpenTelemetry now captures agent traffic at the kernel
&lt;/h2&gt;

&lt;p&gt;OBI attaches kprobes and uprobes in the kernel and parses protocols in a separate process outside your application. Because it reads payloads before encryption, it sees LLM and MCP calls over TLS without injecting code. It deploys as a DaemonSet: one process per node, no restarts.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://opentelemetry.io/docs/zero-code/obi/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;As of OBI v0.9.0&lt;/a&gt;, coverage includes OpenAI, Anthropic, Gemini, Bedrock, Qwen, MCP over JSON-RPC, and embedding and rerank providers. When the &lt;a href="https://github.com/open-telemetry/semantic-conventions-genai?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;GenAI semantic conventions&lt;/a&gt; change, one OBI upgrade updates every service.&lt;/p&gt;

&lt;p&gt;As I said in the talk: "If you cannot observe how many tokens are used, what tools are called, and how many times, it's really hard to figure out how to make your agents better."&lt;/p&gt;

&lt;p&gt;OBI complements SDKs. It captures LLM and tool calls from outside the application process, plus what SDKs cannot see: TLS, DNS, and retries. SDKs add in-process detail OBI cannot observe, such as business spans and custom attributes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Consistent telemetry is a good starting point that still lacks causal context
&lt;/h2&gt;

&lt;p&gt;Consistent telemetry means two things. Every service reports GenAI calls with the same attributes and semantic convention version, and trace context connects those calls across services and languages. In my own production environment, a Go MCP server calls another Go service, which calls a Python service, which calls Gemini or, through OpenRouter, an Anthropic model. One OBI process per node instruments all of it the same way.&lt;/p&gt;

&lt;p&gt;Consistent telemetry still leaves a gap. A trace shows an agent called a tool server dozens of times and slowed down. It does not say why. Answering that takes causal context: a model of how the agent, its tool servers, model providers, and the infrastructure beneath them can fail, and which symptoms each failure produces.&lt;/p&gt;

&lt;p&gt;That is the layer we build at Causely. It uses OBI alongside the metrics, logs, traces, and infrastructure data teams already collect, and adds a causal model that checks observed symptoms against known cause signatures, including &lt;a href="https://docs.causely.ai/reference/root-causes/?subcategory=AI+Model&amp;amp;ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;AI model-specific causes&lt;/a&gt;, and returns a diagnosis.&lt;/p&gt;

&lt;p&gt;This builds on &lt;a href="https://www.causely.ai/blog/why-genai-applications-are-hard-to-operate?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;why GenAI applications are hard to operate&lt;/a&gt; and our earlier work &lt;a href="https://www.causely.ai/blog/auto-instrumenting-java-with-ebpf-no-jvm-changes-required?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;instrumenting Java with OBI&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do next
&lt;/h2&gt;

&lt;p&gt;Watch the full talk below, then deploy &lt;a href="https://opentelemetry.io/docs/zero-code/obi/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;OBI&lt;/a&gt; as a DaemonSet on a cluster running your agents. The project is active and welcomes contributors: join the SIG calls or pick up an open issue in the &lt;a href="https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;OBI repository&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/Me88RYQ3Djg" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

</description>
      <category>opentelemetry</category>
      <category>ai</category>
      <category>observability</category>
      <category>devops</category>
    </item>
    <item>
      <title>How to Operate AI Agents in Production: The Three-Layer Stack</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 05 Oct 2026 16:35:41 +0000</pubDate>
      <link>https://dev.to/causely/how-to-operate-ai-agents-in-production-the-three-layer-stack-4bl4</link>
      <guid>https://dev.to/causely/how-to-operate-ai-agents-in-production-the-three-layer-stack-4bl4</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/how-to-operate-ai-agents-in-production-the-three-layer-stack" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/yyemini/" rel="noopener noreferrer"&gt;Yotam Yemini&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt;: Running agents in production is a layered discipline. Instrumentation captures what the agent did. Eval and observability platforms like Comet, Langfuse, Braintrust, Galileo, and Arize measure whether the output was good. A causal reasoning system explains why behavior changed across the services and infrastructure the agent runs on. Each layer has a distinct job. Together they are what it takes to operate agents in production with confidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  Operating Agents Is Becoming a Layered Discipline
&lt;/h2&gt;

&lt;p&gt;You shipped an agent to production. When its output quality slips, or latency climbs, or it starts failing tasks it used to complete, the first question is always the same: why. Answering it has become its own discipline, and&amp;nbsp;teams&amp;nbsp;running agents at scale are assembling a proper operational stack to do it, the same way teams running microservices did a decade ago.&lt;/p&gt;

&lt;p&gt;Here is how that stack is taking shape, and where a causal reasoning system fits into it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer One: Instrumentation
&lt;/h2&gt;

&lt;p&gt;Everything starts&amp;nbsp;with&amp;nbsp;capturing what the agent actually did. OpenTelemetry's&amp;nbsp;GenAI semantic conventions give you a common vocabulary for LLM calls, tool calls, token usage, and, increasingly, whole agent and MCP call trees as first-class spans. The conventions are still under active development as of mid-2026, so attribute names can&amp;nbsp;shift&amp;nbsp;and you should budget for the occasional rename. But they have&amp;nbsp;consolidated&amp;nbsp;enough to&amp;nbsp;standardize on, and real backends already emit and ingest them. Instrumenting against them gives you a structured, correlated record of every interaction, the foundation the other layers build on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer Two: Evaluation and Agent Observability
&lt;/h2&gt;

&lt;p&gt;Once you have the signals, you need to know whether the agent is doing&amp;nbsp;a good job. This is where the eval and LLM observability platforms live: Comet, Langfuse, Braintrust, Galileo (now part of Cisco),&amp;nbsp;Arize&amp;nbsp;(now part of Dynatrace), and others. They each have a different emphasis. Some lean toward the offline eval and experimentation loop, others toward production monitoring and tracing, but they share a purpose: scoring outputs, catching regressions, and tracking agent behavior against datasets and guardrails.&lt;/p&gt;

&lt;p&gt;Output quality is only part of the story. This layer is also about understanding the agent's operational health: latency and error rates, along with agent-specific signals like repeated or redundant tool calls, cache efficiency, and token consumption, much of which falls out of the same OTel instrumentation from Layer One.&lt;/p&gt;

&lt;p&gt;If you are operating agents in production, you should be using one of these, or a comparable solution. They answer two important questions: what did the agent do, and is it doing its job well? Every team needs to be able to answer those questions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer Three: Reasoning Across the Stack
&lt;/h2&gt;

&lt;p&gt;The first two layers answer what and whether. The third answers why, and it is the hardest of the three.&lt;/p&gt;

&lt;p&gt;An agent in production sits on top of everything else you run. The LLM call is one hop in a chain that includes the services assembling context, the vector stores and databases serving retrieval, the APIs exposed as tools, the queues moving work between steps, and the compute, network, and storage underneath all of it. When output quality drops or latency climbs, the origin is often somewhere in that chain rather than in the agent itself:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A tool starts returning empty results because a backing service degraded, and the agent compensates by retrying or looping around the gap.&lt;/li&gt;
&lt;li&gt;Retrieval quality slips because an embedding pipeline stalled and the index went stale, so answers get quietly worse while traces look normal.&lt;/li&gt;
&lt;li&gt;p95 latency doubles because a database serving context assembly is under memory pressure, not because the model slowed down.&lt;/li&gt;
&lt;li&gt;A routine deployment to a shared dependency changes a default timeout, and the agent starts hitting intermittent tool failures it never saw before, with nothing in the agent itself having changed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Connecting a change in agent behavior to its origin somewhere below the agent is a distinct problem, and it is genuinely hard, because it means reasoning across the causal relationships between the agent and every layer of the stack it depends on. This is not a new goal. Cross-layer causal analysis has been the direction of travel in observability for years. What is still open is how you actually do it in software, rather than leaning on the tribal knowledge of a few senior engineers and an ever-growing pile of runbooks.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Is the Best Approach for Managing AI Agents in Production?
&lt;/h2&gt;

&lt;p&gt;The industry consensus is clear, and the recent moves make it clearer. Cisco acquiring Galileo and &lt;a href="https://www.dynatrace.com/news/press-release/dynatrace-to-acquire-arize/" rel="noopener noreferrer"&gt;Dynatrace acquiring Arize, at a reported $915M&lt;/a&gt;, are both bets on bringing agent telemetry and infrastructure observability under one roof. Datadog has been building toward the same goal organically. The Dynatrace announcement said the premise plainly: when an AI application's output quality slips, the cause can sit anywhere from the prompt to the model to the infrastructure underneath it. We agree that reasoning across the layers of the stack is the right goal, and it is the premise we started Causely on years ago.&lt;/p&gt;

&lt;p&gt;Where approaches diverge is on how you go from observable symptoms to the specific issue that needs to be addressed. The dominant path has been correlation and machine learning over telemetry, and the recent acquisitions extend it by putting more of that telemetry, agent and infrastructure, in the same place. That is a step forward, and co-located data makes correlation easier. But it also has a ceiling: correlation tells you an eval score dropped around the time a database schema changed, which sometimes points at the real issue and sometimes at a coincidence.&lt;/p&gt;

&lt;p&gt;In an environment with many components changing constantly, separating signal from noise takes a model of how system behaviors propagate across layers: which services feed context into which prompts, which tools depend on which backends, how a degradation in one place surfaces as a symptom somewhere else.&lt;/p&gt;

&lt;p&gt;That model-based approach is the bet Causely makes. Instead of correlating signals after the fact, the system&amp;nbsp;maintains&amp;nbsp;a live causal model of the environment and reasons&amp;nbsp;over&amp;nbsp;it deterministically. Because the reasoning follows the model rather than pattern-matching over history, it does not surface hallucinated root causes, and it&amp;nbsp;points at&amp;nbsp;the specific issue to fix rather than a ranked list of correlated suspects. Across production deployments, customers have seen 63% faster incident resolution and 57% lower operating cost, with zero hallucinated root causes.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Does Causely Pinpoint the Cause of Agent Degradations?
&lt;/h2&gt;

&lt;p&gt;We&amp;nbsp;made a decision&amp;nbsp;from day one to build a system that ingests data from any combination of sources, whether that includes OpenTelemetry, your existing observability platforms, or GenAI instrumentation, and produces a semantically consistent representation of your environment. From it, our system instantiates thousands to tens of thousands of causal models that encode how an issue in one part of your stack causes symptoms in others.&amp;nbsp;All of this causal knowledge is continuously maintained as the environment changes.&lt;/p&gt;

&lt;p&gt;There are two primary patterns for how customers are putting this knowledge to use, which help illustrate how it fits alongside the rest of your stack:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;On Demand.&lt;/strong&gt;&amp;nbsp;An alert fires, Causely answers why. Route existing alerts into Causely, or let your agent query Causely's MCP server as part of an investigation. One call returns signal vs. noise, causal analysis, blast radius, and supporting evidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Proactive.&lt;/strong&gt;&amp;nbsp;Causely surfaces emerging system degradation with causal analysis attached and triggers a background agent to act. The same payload can be sent to any destination you choose before an SLO is&amp;nbsp;violated&amp;nbsp;or an incident is declared.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One customer described the proactive pattern in concrete terms. Causely detected emerging degradations and pushed the causal diagnosis to their background agents, which resulted in twelve agent-initiated pull requests that the team merged after human review. By their account, each change saved their five-person team two to three hours over the month, and that is before counting the incidents they prevented. They framed this benefit as the equivalent of adding a full-time engineer dedicated to keeping performance healthy and preventing incidents.&lt;/p&gt;

&lt;p&gt;The eval layer keeps doing what it does best: output quality, regression testing, guardrails, the&amp;nbsp;development loop. Causely sits underneath and across it, connecting what the agent did to why the surrounding system shaped that behavior. If you are building agents and&amp;nbsp;automated&amp;nbsp;workflows on top of this data, we have a lot to talk about.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Whole Stack, Working Together
&lt;/h2&gt;

&lt;p&gt;No single layer is the whole answer, and none of them replaces the others. Instrumentation tells you what the agent did. Evaluation tells you whether the output was good. Causal reasoning tells you why the behavior changed, across the services and infrastructure the agent runs on. The teams operating agents with the most confidence are the ones running all three.&lt;/p&gt;

&lt;p&gt;If you want to see for yourself,&amp;nbsp;&lt;a href="https://www.causely.ai/try" rel="noopener noreferrer"&gt;try it free&lt;/a&gt;. Or reach me directly via email (&lt;a href="mailto:yyemini@causely.ai"&gt;yyemini@causely.ai&lt;/a&gt;) or on &lt;a href="https://www.linkedin.com/in/yyemini/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does Causely replace my eval platform?&lt;/strong&gt;&amp;nbsp;No. Eval platforms own output quality: scoring, datasets, regression testing, guardrails. Causely pinpoints the cause of a degradation by reasoning across the agent layer together with the various services and infrastructure components the agent depends on. They are complementary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I use&lt;/strong&gt;&amp;nbsp;&lt;strong&gt;Arize&lt;/strong&gt;&amp;nbsp;&lt;strong&gt;or Galileo, what do the acquisitions mean for me?&lt;/strong&gt;&amp;nbsp;Those products now live inside larger observability portfolios. What the&amp;nbsp;acquisitions&amp;nbsp;signal is that agent and infrastructure telemetry are converging. Causely is interoperable with&amp;nbsp;all of&amp;nbsp;these products and will stay that way. We already consume data from Dynatrace, Datadog, OpenTelemetry, and others, and reason across all of it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How does Causely know about my agent workflows?&lt;/strong&gt;&amp;nbsp;Causely builds a topology of your environment from the data sources you already have, including&amp;nbsp;OTel&amp;nbsp;GenAI instrumentation where present. Your&amp;nbsp;agent&amp;nbsp;workflows appear in the causal model alongside the services, data stores, and infrastructure they depend on, so an issue anywhere in the stack can be mapped directly back to the workflows it affects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does "proactive" mean here concretely?&lt;/strong&gt;&amp;nbsp;The system does not wait for you to query it. The causal model is evaluated continuously as telemetry flows in, so emerging degradations are&amp;nbsp;identified&amp;nbsp;and attributed to their cause as they develop. Causely can trigger a background agent, send a message, or&amp;nbsp;initiate&amp;nbsp;a remediation workflow, often before the issue rises to the level of an incident.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>observability</category>
      <category>sre</category>
    </item>
    <item>
      <title>How to make Claude background agents 3-5x faster and cheaper</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Wed, 30 Sep 2026 18:22:26 +0000</pubDate>
      <link>https://dev.to/causely/how-to-make-claude-background-agents-3-5x-faster-and-cheaper-1438</link>
      <guid>https://dev.to/causely/how-to-make-claude-background-agents-3-5x-faster-and-cheaper-1438</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/how-to-make-claude-background-agent-3-5x-faster-and-cheaper" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/byemini/" rel="noopener noreferrer"&gt;Ben Yemini&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;We ran two Claude Managed Agents against injected regressions in a 36-microservice Go app on Kubernetes. Both got the same page, same cluster, Grafana access, and the same source code. One also had Causely's MCP server. Both agents produced a correct fix every time. The agent with causal context used 3.6x to 7x fewer tool calls, finished 3.6x to 5.7x faster, and cost 3.5x to 5x less.&lt;/p&gt;

&lt;p&gt;In my previous posts on Claude Managed Agents I discussed &lt;a href="https://www.causely.ai/blog/is-access-enough-auth-patterns-for-claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;how to give an agent access to production&lt;/a&gt; and &lt;a href="https://www.causely.ai/blog/what-should-trigger-an-autonomous-agent-in-production?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;what should trigger it&lt;/a&gt;. If you have not already read those, here is the most relevant part to get you caught up. &lt;a href="https://claude.com/blog/the-ai-native-sdlc-playbook?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Anthropic's AI-Native SDLC Playbook&lt;/a&gt;, published this August, provides best practices for applying AI across six stages. The six-stage framework used in the playbook should be familiar to anyone who has been working in enterprise software. Stage six, maintain, is where background agents serve a critical role. In this post, we measure the effectiveness of the agent running at this step and the impact of providing the agent with causal context.&lt;/p&gt;

&lt;h2&gt;
  
  
  Testing for one key difference
&lt;/h2&gt;

&lt;p&gt;To understand the impact of causal context, we set up two identical Claude Managed Agents &lt;a href="https://github.com/causely-oss/background-agents/tree/main/claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;building on the framework and repo examples&lt;/a&gt; shared by Anthropic. Both agents can query what is running in a Kubernetes cluster via the k8s MCP. Both have access to Grafana MCP, including application metrics, traces, and logs. And both have access to the application source code on a GitHub branch, where the regression is introduced; more on that below. One also has access to Causely and the causal context it provides, based on the same traces, metrics, and logs that flow into Grafana.&lt;/p&gt;

&lt;p&gt;We ran both agents through three different fault scenarios. And prompted them with the same starting point (e.g., service x is degrading). In each case, the service we named is two to three hops from where the actual fault is introduced. This is used to mimic what we often see in larger environments, where an on-call engineer manually declares an incident but cannot identify the exact service at fault at the start of the investigation. Or in environments where alerts automatically trigger the start of the investigation; however, the triggering alert is just for one of the multiple services whose threshold is out of band.&lt;/p&gt;

&lt;p&gt;Fault scenarios and distance from the service named in the prompt&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Fault&lt;/th&gt;
&lt;th&gt;Service named in prompt&lt;/th&gt;
&lt;th&gt;Hops&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;billing service timeout&lt;/td&gt;
&lt;td&gt;api-gateway&lt;/td&gt;
&lt;td&gt;2: api-gateway → checkout → billing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;recommendation service memory limits&lt;/td&gt;
&lt;td&gt;search&lt;/td&gt;
&lt;td&gt;2: search → ranking → recommendation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;pricing service excessive calls&lt;/td&gt;
&lt;td&gt;frontend&lt;/td&gt;
&lt;td&gt;3: frontend → api-gateway → cart → pricing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For each scenario we measure the correctness of the fix by reviewing the PR the agent proposes, the tool calls, the time it takes and the cost.&lt;/p&gt;

&lt;p&gt;The one other difference is that the agent with access to Causely is instructed to first call &lt;code&gt;get_issues&lt;/code&gt; scoped to the namespace where the app is running. This is similar to triggering the agent with an Issue, which is what we previously argued is much more effective than triggering it with an out-of-band threshold violation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Typical regressions as faults
&lt;/h2&gt;

&lt;p&gt;To make these scenarios more real-world, we introduced code-based regressions in either the application or the infrastructure configuration, rather than relying on fault-injection methodologies. The application itself is an e-commerce app composed of 36 microservices written in Go, running in a single Kubernetes namespace, with Kafka, Redis, and Postgres for persistence. A frontend and a fan-out to multiple services, with a load generator running continuously to mimic eight different user flows. The repo, including app source code and scenario setup, is &lt;a href="https://github.com/causely-oss/regression-lab?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;available here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Here are the three regression fault scenarios:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Billing Service Timeout
&lt;/h3&gt;

&lt;p&gt;The billing service's HTTP outbound client has been refactored to improve the use of existing connections. The refactor accidentally removes the connection request timeout. Now, when the payment-adapter service is slow, requests from the billing service hang instead of failing fast, and checkout latency starts to climb.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Recommendation Service Memory Limits
&lt;/h3&gt;

&lt;p&gt;An inadvertent change to a YAML file set the request memory to 8 Mi and the limits to 10 Mi, for a service that typically runs at ~10 to 13 Mi. The pods start experiencing OOMKilled restarts; multiple services (e.g., search, ranking) experience elevated error rates, and search latency climbs as ranking waits on search.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Pricing Service Excessive Calls
&lt;/h3&gt;

&lt;p&gt;A change to the pricing service code now calls the discount service once per line item in the cart, rather than once per cart. Both the discount and cart service experiences increase latency as cart size grows, and frontend latency increases for users trying to complete a transaction.&lt;/p&gt;

&lt;p&gt;Each of these regressions is introduced in a branch that includes additional unrelated changes, so a simple revert of the latest code changes is not an effective remediation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Searching vs. starting with a diagnosis
&lt;/h2&gt;

&lt;p&gt;With causal context, the agent takes a different approach. Instead of rediscovering the system from scratch each time, it starts from an answer and focuses on confirming its validity and fixing the underlying code regression. This can be seen by reviewing the session traces. The Claude Managed Agents platform comes with this visibility. You can see every call the agent makes, how long each takes, where they encounter errors, and how many input or output tokens they consume.&lt;/p&gt;

&lt;p&gt;Reviewing the Baseline agent – the one without causal context – we can see the following pattern. The agent first orients on the environment by listing pods in the namespace, checking for recent k8s events, and pulling logs for the named service. It then goes deep into the codebase to understand the layout and reconstruct the service dependency graph. It now starts a broad trial-and-error telemetry query for each service in the dependency chain, using dozens of PromQL queries. It is essentially making educated guesses about which services and signals matter and pursuing those guesses. At the same time, it has a side-quest running checking for firing alerts, autoscaling config, and additional infrastructure signals. Once it has the suspected service, it focuses on git logs to understand recent commits and develop a hypothesis for what the exact app or infra code change needs to be. Finally, it confirms that current live metrics support this hypothesis, applies the edits, commits, and pushes.&lt;/p&gt;

&lt;p&gt;On the other hand, the agent with causal context starts by asking, “What is wrong?” The first call is almost always &lt;code&gt;get_issues&lt;/code&gt;. This returns the implicated entity. The agent now investigates this lead to corroborate Causely’s diagnosis with raw evidence. It then focuses on the specific file(s) associated with the flagged entity rather than reviewing the entire repo and reconstructing the dependency graph. From there, it follows a similar pattern: reviewing the git history for this part of the codebase, then following the same closing sequence as the baseline agent – fix, commit, and push.&lt;/p&gt;

&lt;h2&gt;
  
  
  Both agents were correct, but one did 5x the work
&lt;/h2&gt;

&lt;p&gt;In all the scenarios we tested, both agents identified the correct underlying cause and raised a PR with a valid fix. Neither blamed the service named in the initial prompt (e.g., search in fault scenario 2). The difference was speed and cost. The agent without causal context was slower, made many more tool calls, and used more tokens.&lt;/p&gt;

&lt;p&gt;One run per scenario: baseline versus an agent with Causely&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Fault scenario&lt;/th&gt;
&lt;th&gt;Tool calls (Baseline)&lt;/th&gt;
&lt;th&gt;Tool calls (With Causely)&lt;/th&gt;
&lt;th&gt;Time (Baseline)&lt;/th&gt;
&lt;th&gt;Time (With Causely)&lt;/th&gt;
&lt;th&gt;Cost (Baseline)&lt;/th&gt;
&lt;th&gt;Cost (With Causely)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;One&lt;/td&gt;
&lt;td&gt;155&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;18m 1s&lt;/td&gt;
&lt;td&gt;3m 10s&lt;/td&gt;
&lt;td&gt;$14.34&lt;/td&gt;
&lt;td&gt;$3.17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Two&lt;/td&gt;
&lt;td&gt;125&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;12m 18s&lt;/td&gt;
&lt;td&gt;2m 28s&lt;/td&gt;
&lt;td&gt;$10.77&lt;/td&gt;
&lt;td&gt;$2.17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Three&lt;/td&gt;
&lt;td&gt;105&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;10m 31s&lt;/td&gt;
&lt;td&gt;2m 57s&lt;/td&gt;
&lt;td&gt;$11.92&lt;/td&gt;
&lt;td&gt;$3.39&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The table shows one run per scenario. Managed Agent sessions vary from run to run in tool calls, time, and overall cost. The size of the gap is more informative than the exact values. A 5x difference in cost (fault scenario two) or a 7x difference in tool calls (fault scenario one) will not close to 1x with additional runs.&lt;/p&gt;

&lt;p&gt;Without causal context, the agent rebuilds the system picture on every run, making many more tool calls and burning tokens to reconstruct what Causely already exposes as its &lt;a href="https://www.causely.ai/blog/semantics-in-observability?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;semantics&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ready to try this with your agent?
&lt;/h2&gt;

&lt;p&gt;We’ve made the scenarios and evaluation harness available in our Causely open-source repos. The setup for the agents and evaluation approach can be found in the background agents repo under &lt;a href="https://github.com/causely-oss/background-agents/tree/main/claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Claude Managed Agents&lt;/a&gt;, and the application, including code regression faults, can be found under the &lt;a href="https://github.com/causely-oss/regression-lab?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Regression Lab&lt;/a&gt; repo.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>claude</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>How to Check an Agent's Diagnosis Before It Touches Production</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 21 Sep 2026 20:53:24 +0000</pubDate>
      <link>https://dev.to/causely/how-to-check-an-agents-diagnosis-before-it-touches-production-fp6</link>
      <guid>https://dev.to/causely/how-to-check-an-agents-diagnosis-before-it-touches-production-fp6</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/how-to-check-an-agents-diagnosis-before-it-touches-production" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/byemini/" rel="noopener noreferrer"&gt;Ben Yemini&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;When teams get ready to add agents to their on-call workflows, the hard part is deciding when to let those agents act on production versus just advise. A root cause claim only earns that trust if you can verify what the agent ruled out. Causely's causal model now exposes that reasoning directly to agents: which alternative diagnoses were considered for a symptom, what evidence ruled out those alternatives, and how far the failure would have spread if left unaddressed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Acting vs. Advising
&lt;/h2&gt;

&lt;p&gt;A recent &lt;a href="https://www.reddit.com/r/kubernetes/comments/1vq3ot9/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;r/kubernetes thread&lt;/a&gt; separated a remediation agent into two parts: an LLM proposes actions such as scaling, rolling back, cordoning, or draining, and a separate deterministic layer vetoes anything unsafe against the live cluster state before it executes. The design assumes a proposal and the decision to act on it are two different things.&lt;/p&gt;

&lt;p&gt;The top reply doesn't argue with the architecture. It points to the actual gap: confidence in the cause and the safety to act are two different things, and they often get conflated. The suggested fix is to force the agent to explain why this diagnosis, specifically, before it hands over a proposed action. That way a human or a policy engine has something to review, not just a recommendation to accept or reject. Another engineer in the same thread pushes further: a veto needs memory. Blast-radius budgets per namespace, cooldowns, and a rule to halt and page a human if the expected effect doesn't show up. And the veto must check real cluster state itself, not trust what the agent reports about that state.&lt;/p&gt;

&lt;p&gt;Platform teams everywhere are running into the same challenge: when is an agent's diagnostic capability good enough to let it act in production?&lt;/p&gt;

&lt;h2&gt;
  
  
  Why isn't "the agent is confident" the same as "the agent is right"?
&lt;/h2&gt;

&lt;p&gt;An agent proposing an action and an agent explaining its diagnosis are two different things. Only one of them is checkable before anyone touches production. A confidence score is an assertion. A list of possible explanations for the symptom, with the causal chain behind each, is something a reviewer — human or automated — can inspect.&lt;/p&gt;

&lt;p&gt;That's the same problem the thread's second engineer described: a veto that trusts the agent's self-report is a second opinion, not a check. A veto that can see what the model considered and rejected, and how large the blast radius of the winning diagnosis is, has something real to validate with.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does it mean to ask an agent "why this and not that"?
&lt;/h2&gt;

&lt;p&gt;When it comes to understanding cause and effect, most people familiar with the domain don't usually ask, "why did X happen?" They tend to ask "why X, rather than Y," a finding from explainable-AI research on &lt;a href="https://arxiv.org/pdf/1706.07269?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;contrastive explanation&lt;/a&gt;: an explanation is only satisfying relative to some alternative that didn't occur, even when that alternative is never stated out loud. &lt;a href="https://arxiv.org/html/2507.08454?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;A 2025 formalization of the problem&lt;/a&gt; frames it directly as "why P but not Q" and treats computing the difference between the two as the actual explanation task, not as an afterthought.&lt;/p&gt;

&lt;p&gt;Causely's MCP server now exposes two tools designed specifically for this question. &lt;code&gt;get_potential_diagnoses&lt;/code&gt; tells you what could explain a given symptom, not just the one the causal model selected, and shows the causal chain behind each. &lt;code&gt;get_signal_potential_diagnoses&lt;/code&gt; runs the same question in the other direction: given an observed signal, what could explain it, and what's the causal chain connecting each candidate to that signal? Both draw on the same causal model that produces the primary diagnosis: the same reasoning, made visible instead of collapsed into a single answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Here is an example
&lt;/h2&gt;

&lt;p&gt;Take a familiar incident: elevated latency and timeouts on a checkout service, several layers downstream from a shared database connection pool. An LLM-only agent traces this from the symptom alone. It queries telemetry, traces back from the checkout service, and finds a CPU spike on an adjacent service along the way. It diagnoses the CPU spike as the cause and restarts that service. The dashboard improves. The agent treats the incident as resolved.&lt;/p&gt;

&lt;p&gt;The real cause, connection exhaustion on the shared database connection pool upstream, provides no direct signal at that layer, so the agent never sees it. The restart bought time by clearing the CPU spike, not by fixing the underlying problem. The incident resurfaces when the connection limit is reached again, and the agent repeats this loop.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5bnqdpeo2vm0t2k1lmhh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5bnqdpeo2vm0t2k1lmhh.png" width="797" height="422"&gt;&lt;/a&gt;Example of Claude explaining why this diagnosis and what else was considered but ruled out. &lt;/p&gt;

&lt;p&gt;Calling &lt;code&gt;get_signal_potential_diagnoses&lt;/code&gt; on the original symptom surfaces the connection-exhaustion diagnosis alongside the CPU-spike alternative, each with the causal chain showing what it explains and what it fails to account for. The CPU spike doesn't explain the downstream timeout pattern; the connection exhaustion does. That comparison is what makes the diagnosis reviewable before a remediation agent acts on it, not just the final answer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frzjf1j3t1t1f42skpvmv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frzjf1j3t1t1f42skpvmv.png" width="800" height="724"&gt;&lt;/a&gt;Example of Claude visualizing the causal chain for this diagnosis&lt;/p&gt;

&lt;h2&gt;
  
  
  How far would this failure have spread if it wasn't caught?
&lt;/h2&gt;

&lt;p&gt;A diagnosis with a small, well-understood blast radius and a diagnosis with a large, uncertain one are different risk decisions, even if a model is equally confident in both. &lt;code&gt;rank_entities&lt;/code&gt; and &lt;code&gt;get_diagnosis_observable_signals&lt;/code&gt; surface which entities the failure would affect and which signals to expect downstream before any action is taken.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Finnogyowoh7ay0qhq1bj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Finnogyowoh7ay0qhq1bj.png" width="772" height="514"&gt;&lt;/a&gt;Example of get_diagnosis_observable_signals tool call response in Claude Desktop&lt;/p&gt;

&lt;p&gt;This is the concrete version of the veto idea from the above Kubernetes thread: a blast-radius budget is only useful if something can compute the blast radius first. Ranking entities by dependency exposure is what gives a policy engine, or a human running the on-call rotation, a number to set that budget against instead of a guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where can causal explanations go further?
&lt;/h2&gt;

&lt;p&gt;Today's tools answer "why this and not that" and "how far would it spread" against the current state of the causal model. The next layer is forward simulation: "what would break if this component degraded," asked before any symptom appears, not after. That requires modeling failure modes as first-class entities with their own preconditions and blast radius, rather than only reasoning from observed anomalies backward. It's a natural extension of the same causal model: the same questions, asked earlier instead of after the fact.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is a contrastive explanation in root cause analysis?&lt;/strong&gt; Contrastive explanation answers "why this cause, not that one" instead of just "why did this happen." Research on explainable AI has found that people evaluate explanations relative to an alternative that didn't occur, even if that alternative is unstated. In root cause analysis, this means showing which diagnoses were considered and ruled out, not only the one that was selected.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How is this different from a deterministic safety veto layer?&lt;/strong&gt; A safety veto checks whether a proposed action is safe to execute against live system state. Contrastive diagnosis explanation checks whether the reasoning behind the diagnosis holds up, and which alternatives were ruled out and on what evidence. The two are complementary: a veto layer is more useful with a diagnosis it can inspect, rather than a bare confidence score.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I use the new MCP tools to check an agent's diagnosis before it acts?&lt;/strong&gt; Call &lt;code&gt;get_potential_diagnoses&lt;/code&gt; or &lt;code&gt;get_signal_potential_diagnoses&lt;/code&gt; against the symptom an agent is reasoning to see what could explain it and the causal chain behind each candidate. Pair this with &lt;code&gt;rank_entities&lt;/code&gt; or &lt;code&gt;get_diagnosis_observable_signals&lt;/code&gt; to see the blast radius of the leading diagnosis before approving any remediation action.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Doesn't surfacing alternative diagnoses just mean the model is less confident?&lt;/strong&gt; No. Showing what could explain a symptom and the causal chain behind each candidate makes a diagnosis checkable, not uncertain. A model that only reports its top answer is asking to be trusted. A model that shows what else it considered and why each was ruled out is being reviewed. The second is the stronger claim.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Read &lt;a href="https://www.causely.ai/blog/how-causal-reasoning-addresses-the-limitations-of-llms-in-observability?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;How Causal Reasoning Addresses the Limitations of LLMs in Observability&lt;/a&gt; for the causal model this builds on.&lt;/li&gt;
&lt;li&gt;Read &lt;a href="https://www.causely.ai/blog/beyond-the-blast-radius-demystifying-and-mitigating-cascading-microservice-issues?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Beyond the Blast Radius&lt;/a&gt; for more on dependency-aware impact analysis.&lt;/li&gt;
&lt;li&gt;Learn more about &lt;a href="https://docs.causely.ai/agent-integration/mcp-server/?ref=causely-blog.ghost.io#full-tool-reference" rel="noopener noreferrer"&gt;Causely MCP tools&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>kubernetes</category>
      <category>observability</category>
    </item>
    <item>
      <title>What should trigger an autonomous agent in production?</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 14 Sep 2026 21:51:17 +0000</pubDate>
      <link>https://dev.to/causely/what-should-trigger-an-autonomous-agent-in-production-3ado</link>
      <guid>https://dev.to/causely/what-should-trigger-an-autonomous-agent-in-production-3ado</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/blog/what-should-trigger-an-autonomous-agent-in-production" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/byemini/" rel="noopener noreferrer"&gt;Ben Yemini&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;Trigger an autonomous agent on a diagnosis. Anthropic's AI-native SDLC playbook closes the loop by using a deterministic script to monitor production and start a Claude session when a control band is breached. That works, but it also means the agent spends its first and most expensive phase working out what broke. A trigger that sets off a causal chain carries it further along.&lt;/p&gt;

&lt;p&gt;Anthropic published &lt;a href="https://claude.com/blog/the-ai-native-sdlc-playbook?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;The AI-Native SDLC Playbook&lt;/a&gt; last month. It walks through six stages, from plan through maintain, and describes what each one looks like when agents do the work, and people stay accountable for the decisions. Most of the attention (and value delivered) by large language models so far has gone to Build. But Stage six, Maintain, is the more interesting one, and it’s the one that turns the line into a loop. It is also the one worth reading closely if you run production.&lt;/p&gt;

&lt;p&gt;In our last post, we set up &lt;a href="https://www.causely.ai/blog/is-access-enough-auth-patterns-for-claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Claude managed agents against a local Kubernetes cluster&lt;/a&gt;, gave them Grafana and Causely MCP tools, and worked through the authentication pattern for each. This post is about the trigger because it sets up the agent for everything it does afterward.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stage six
&lt;/h2&gt;

&lt;p&gt;Stage six closes the loop with a script that monitors production and invokes Claude when a metric falls outside its expected range. The playbook picks one metric with a rolling baseline, applies Western Electric rules so that slow drift registers alongside spikes, and selects the next step based on how far the metric strays. At 1σ, the script logs. At 2σ, Claude is invoked read-only to diagnose. At 3σ, Claude may act only by opening a PR into the review gate or by triggering a pre-approved runbook. I hope I have not lost those of you who dislike statistics.&lt;/p&gt;

&lt;p&gt;The agent writes what it finds as an intent.md in the Stage one format, and the finding re-enters the pipeline like any other piece of work. Nobody starts it.&lt;/p&gt;

&lt;p&gt;The detection half stays out of the model's hands. Louis Claxton, writing for Anthropic, is direct about it: &lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"detection stays entirely deterministic, with no model involved." &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That constraint is the right one. A trigger that fired because a language model felt uneasy is not a trigger you can audit, tune, or explain to a change board.&lt;/p&gt;

&lt;h2&gt;
  
  
  A band breach carries one fact
&lt;/h2&gt;

&lt;p&gt;A band breach tells you that a metric left its range. The affected entity, which symptoms are consequences of which, and how far the problem reaches, all must be reconstructed before the agent can decide anything. That reconstruction is the job for the diagnosis.&lt;/p&gt;

&lt;p&gt;The other challenge with this approach is that each band watches a single metric, so you need a detection script for every failure mode you anticipate. In a cluster of dependent services, degradation spreads across many entities at once and often breaches nothing decisively on any single one. Every service looks slightly worse than usual. No script fires, and the loop never starts.&lt;/p&gt;

&lt;h2&gt;
  
  
  A diagnosis carries the causal chain behind it
&lt;/h2&gt;

&lt;p&gt;A Causely &lt;a href="https://www.causely.ai/blog/introducing-issues-one-problem-one-thread-start-to-finish?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Issue&lt;/a&gt; names the affected entity, designates a primary diagnosis, and exposes the chain of evidence behind it. The agent opens by explaining what could explain the symptom and why, rather than deriving that from the raw cluster state.&lt;/p&gt;

&lt;p&gt;Causely computes this from the instrumentation you already run. The causal model encodes, ahead of time, which root causes each type of entity can have and which observable &lt;a href="https://docs.causely.ai/reference/symptoms/?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;signals&lt;/a&gt; each one produces, on that entity and on the entities that depend on it. Faulty error handling on an RPC method, a slow consumer on a topic, a saturated connection pool on a database client: each has a known signature. A slow consumer shows up as growing queue depth on the topic and rising latency on whatever is waiting for the result. Faulty error handling shows up as a high error rate on that method and on every service that calls it.&lt;/p&gt;

&lt;p&gt;At runtime, metrics, alerts, and events are mapped to entities and become observed signals. The model then asks, for the set of currently active signals, which diagnosis best explains them. Because the signatures are encoded in advance, this is a lookup against the model, not a search through raw telemetry. What arrives in the Issue is that diagnosis and the chain of observed signals that supports it, not a shorter list of alerts.&lt;/p&gt;

&lt;p&gt;This inference is deterministic. The same signals on the same topology produce the same diagnosis. That matters for Stage six: the causal layer can sit on the detection side of the playbook's line without breaking the rule that no model participates in detection.&lt;/p&gt;

&lt;h2&gt;
  
  
  Here is an example: a Causely Issue becomes a pull request with nobody in the loop
&lt;/h2&gt;

&lt;p&gt;Causely posts an Issue notification to a webhook receiver you host. The receiver turns it into the agent's first message and starts a session. The receiver is a small FastAPI app in our &lt;a href="https://github.com/causely-oss/background-agents/tree/main/claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;example repository&lt;/a&gt;. The &lt;a href="https://platform.claude.com/docs/en/managed-agents/overview?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Managed Agents API&lt;/a&gt; has no inbound webhook of its own.&lt;/p&gt;

&lt;p&gt;Here is one session end-to-end. Payments are failing in a cluster: checkout returns errors, billing logs timeouts, and the frontend is slow. Nobody has been paged.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;18:14:03.&lt;/strong&gt; Causely raises an Issue on &lt;code&gt;payment-adapter&lt;/code&gt;, severity Critical: calls to &lt;code&gt;external-payment-api:8115&lt;/code&gt; are timing out and being canceled. The receiver forwards it. The agent's first message carries the affected entity, the diagnosis, the evidence, and the Issue id, with an instruction to call &lt;code&gt;get_issue_details&lt;/code&gt; before anything else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;18:14:05.&lt;/strong&gt; The agent makes that call. Causely returns the causal chain. The external payment API is slow to accept charges. Payment-adapter holds the caller's request open while it waits. Checkout's 5s client timeout fires and cancels the whole chain, so the errors land three hops from the cause: 796 canceled calls on payment-adapter, 500 failed calls from billing-service, 657 timeouts on checkout, five services affected. Causely's suggested remediation is attached: add circuit-breaking in the payment-adapter so a slow dependency doesn't take checkout down with it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;18:14:19.&lt;/strong&gt; The agent has the root cause, the evidence, and the blast radius. Sixteen seconds. It moves to the fix.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;18:14:20 to 18:18:57.&lt;/strong&gt; Everything from here happens inside the repository. The agent reads&lt;code&gt;payment-adapter/main.go&lt;/code&gt;, finds the outbound call, edits, reviews its own diff, commits, pushes, and opens the pull request. It never goes back to Causely, Grafana, or Kubernetes. It already knows what it is fixing and why.&lt;/p&gt;

&lt;p&gt;The pull request adds a circuit breaker and a per-call deadline around the external payment call, 87 lines in one file. The description explains why this fix, and not another: the external API is outside the repository, the cascade is inside it, and stopping the cascade restores checkout. Causely's diagnosis and evidence are in the PR body for the reviewer.&lt;/p&gt;

&lt;p&gt;Four minutes and fifty-four seconds from Issue to pull request. Seventeen seconds of that was diagnosis. The first human involvement is the review.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi9dh2um1cdq65kywixb1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi9dh2um1cdq65kywixb1.png" alt="Example managed agent being triggered by Causely Issue notification" width="799" height="343"&gt;&lt;/a&gt;&lt;em&gt;Example managed agent being triggered by Causely Issue notification&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Under a threshold trigger, the same session would have opened on "checkout error rate is high" and spent its first minutes listing services, pulling error rates, and reading logs to discover that the payment-adapter was where the cancellations originated and that the external API was why. Here, that work was done deterministically by the causal model before the session started. The agent's tokens are spent on the code fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is a control band in the AI-native SDLC?&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
A control band is the expected range for a production metric, defined by a deterministic script rather than by a model. Anthropic's AI-native SDLC playbook sets bands from a rolling baseline and tiers the response by how far the metric strays: log at 1σ, invoke an agent read-only to diagnose at 2σ, allow a pull request or a pre-approved runbook at 3σ.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the difference between triggering an agent on an alert and triggering it on a diagnosis?&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
An alert reports that one signal left its range. A diagnosis names the affected entity and exposes the causal chain that explains the symptom. An agent, triggered by an alert, reconstructs the chain from the raw cluster state before it can act. An agent, triggered by a diagnosis, starts from that diagnosis and spends its budget on the fix.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I trigger a Claude managed agent from an external system?&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
Host a receiver yourself. The Managed Agents API supports cron and manual deployment triggers, and its webhook helper handles outbound notifications about session state, so there is no inbound webhook you can point a monitoring tool at. Run a small HTTP service that authenticates the payload, turns it into the agent's first message, and calls the sessions API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I keep an autonomous agent from investigating everything?&lt;/strong&gt;&lt;br&gt;&lt;br&gt;
Filter at the source. Send only Critical and High severity notifications, and send Issues rather than individual defects, so that one incident produces one investigation instead of several. Every session costs tokens and runs without supervision, so the severity filter does double duty as a cost control and a noise control.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;If the agent starts from a diagnosis, who checks the diagnosis?&lt;/strong&gt; The reviewer on the pull request, using the causal chain the diagnosis carries. Causely exposes what could explain the symptom and the evidence for each candidate, so a person or a second agent can ask why this diagnosis rather than an alternative before any change is made. Branch protection keeps the agent from approving its own work.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do next
&lt;/h2&gt;

&lt;p&gt;Clone the &lt;a href="https://github.com/causely-oss/background-agents/tree/main/claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;example repository&lt;/a&gt; and point a Causely Issue notification at a managed agent. The Kubernetes half runs against a local kind cluster, so you can see a session start from a notification before you connect anything real.&lt;/p&gt;

&lt;p&gt;Next Claude Managed Agents post: multiple scenarios run with and without causal context, measuring tokens, time to diagnosis, and whether the pull request the agent opened fixed the fault.&lt;/p&gt;

</description>
      <category>agents</category>
      <category>sre</category>
      <category>ai</category>
      <category>backgroundagents</category>
    </item>
    <item>
      <title>How Open-Inspect and Causely close the loop on autonomous remediation</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Thu, 27 Aug 2026 20:36:29 +0000</pubDate>
      <link>https://dev.to/causely/how-open-inspect-and-causely-close-the-loop-on-autonomous-remediation-16jp</link>
      <guid>https://dev.to/causely/how-open-inspect-and-causely-close-the-loop-on-autonomous-remediation-16jp</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://www.causely.ai/" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/enlinxu/" rel="noopener noreferrer"&gt;Enlin Xu&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;Open-Inspect gives a background agent somewhere to run: a full development environment, triggered by an event, with no human at the keyboard. Causely gives that agent a causal diagnosis instead of a pile of telemetry to interpret. The Cloud &amp;amp; Platform Operations team at Fountain wired both together for proactive production ops, and the pattern is now documented in a runnable example.&lt;/p&gt;

&lt;p&gt;On June 12, 2026, we ran a live session with Cole Murray, creator of Open-Inspect, and Martin Roberts, Cloud &amp;amp; Platform Operations lead at Fountain. The topic was proactive remediation: agents that fix issues before anyone notices, not agents that wait for a page. &lt;a href="https://www.youtube.com/live/L0BXwymeYvk?t=1289&amp;amp;si=0sPQwwzOa-SpY74I&amp;amp;ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Watch the recording&lt;/a&gt; for Martin's walkthrough of the production setup. This post provides more detail on the mechanics behind this setup and includes a link to a working example.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does Open-Inspect run?
&lt;/h2&gt;

&lt;p&gt;Open-Inspect is an open-source background coding agent, &lt;a href="https://github.com/ColeMurray/background-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;built by Cole Murray&lt;/a&gt; and modeled on Ramp's internal Inspect tool. It gives an agent a full development environment (Node.js, Python, git, browser automation, VS Code) and lets that agent pick up work from a webhook, a cron schedule, a Slack message, or a monitoring alert, then run unattended until the task is done.&lt;/p&gt;

&lt;p&gt;Two important aspects of Open-Inspect are relevant to ops use cases. First, sessions persist: an agent can run for minutes or hours and be re-triggered later on the same thread, which fits an investigation that doesn't resolve in one shot. Second, the security model is single-tenant by design. Every user in a deployment shares the same GitHub App installation and repository access. That works for a trusted internal team where everyone already has access to the same repos. Everyone sharing that deployment shares the same GitHub App access. Organizations with multiple teams that need separate boundaries can run separate deployments, one per team, rather than pooling everyone into one.&lt;/p&gt;

&lt;p&gt;Open-Inspect decides where an agent runs and what it's allowed to touch. It has no opinion on why and when to act. This is the context it needs to receive from another source.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does Causely add to a background agent?
&lt;/h2&gt;

&lt;p&gt;An agent with a working development environment and repository access still must be told what to act on. It's similar to the point we made in our blog about &lt;a href="https://www.causely.ai/blog/is-access-enough-auth-patterns-for-claude-managed-agents?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Claude Managed Agents&lt;/a&gt;: an agent that can reach every system in your environment still isn't one you'd trust to act on a guess.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://docs.causely.ai/agent-integration/mcp-server?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Causely MCP server&lt;/a&gt; gives your agent a specific, mechanistic diagnosis, such as Postgres idle-in-transaction accumulation or Redis cache miss storm, that is distinct from the symptoms it produces. A peer-reviewed study we published showed that agents working from Causely's causal context reached a diagnosis 2-3x faster than agents reasoning over raw telemetry.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does Causely make agents faster?
&lt;/h2&gt;

&lt;p&gt;The reason agents leveraging Causely diagnose problems faster is that they never need to rebuild the environment's dependency structure at query time. Not only dependencies between entities in the environment, but also the symptoms that may be observed and the potential causes that could produce those symptoms given the topology at that moment in time. Read the &lt;a href="https://www.causely.ai/blog/your-ai-ops-agent-is-guessing?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;full benchmark study&lt;/a&gt; for the methodology here.&lt;/p&gt;

&lt;h2&gt;
  
  
  How does the integration work?
&lt;/h2&gt;

&lt;p&gt;Open-Inspect answers where and how an agent can act. Causely answers when and what it should act on. An agent with both can efficiently move from an issue to a verified fix without a human bridging the gap.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpsnta5jz9kj5ytphn78a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpsnta5jz9kj5ytphn78a.png" alt="Causely and Open-Inspect data flow" width="680" height="720"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Causely notification triggers the agent to Open PR&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Causely runs continuously over the environment, ingesting telemetry locally and maintaining a semantic understanding of the system. When it identifies an issue, a webhook fires carrying both the issue and the diagnosis. That notification routes to the specific background agent with repo access for that environment. The agent calls &lt;code&gt;get_issue_details&lt;/code&gt; to retrieve the causal context behind it and other Causely MCP tools, like &lt;code&gt;get_logs&lt;/code&gt;, if it needs more evidence. Then it opens a PR.&lt;/p&gt;

&lt;p&gt;The agent never starts from a raw alert. The webhook already carries a diagnosis, not a symptom. &lt;code&gt;get_issue_details&lt;/code&gt; and &lt;code&gt;get_logs&lt;/code&gt; add depth to that diagnosis. They don't build it from scratch. The output is a PR, and a human reviews the fix to verify the change before it ships.&lt;/p&gt;

&lt;p&gt;Fountain runs this exact sequence: webhook, &lt;code&gt;get_issue_details&lt;/code&gt;, &lt;code&gt;get_logs&lt;/code&gt; if needed, PR. An engineer reviews and merges from there. Martin described fixing issues before the traditional alert would have fired, let alone before any end-user escalation:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"An agent works right away. It receives the request and, within seconds, identifies the root cause with help from the Causely MCP tool. Within minutes it has the pull request ready. That is much, much faster than how any ops team can function while working on the problem".&lt;br&gt;
— Martin Roberts, Cloud &amp;amp; Platform Operations lead at Fountain&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What to do next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Watch &lt;a href="https://www.youtube.com/live/L0BXwymeYvk?t=216s&amp;amp;ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;the full webinar&lt;/a&gt; for Martin's production walkthrough and the live Q&amp;amp;A.&lt;/li&gt;
&lt;li&gt;Read &lt;a href="https://github.com/causely-oss/background-agents/tree/main/open-inspect?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;the causely-oss example&lt;/a&gt; and adapt the trigger pattern to your own alerts.&lt;/li&gt;
&lt;li&gt;If you're building a custom agent instead, start with &lt;a href="https://www.causely.ai/blog/causely-mcp-skills-stop-prompt-engineering-your-reliability-agent?ref=causely-blog.ghost.io" rel="noopener noreferrer"&gt;Causely MCP Skills&lt;/a&gt; for the tool-selection logic Open-Inspect's example already uses.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What's the difference between Open-Inspect and Causely?&lt;/strong&gt; Open-Inspect is a background agent execution system: it decides where an agent runs and what it's allowed to touch. Causely is a causal reasoning system: it determines what issues need to be acted on and when. Open-Inspect provides the environment; Causely provides the diagnosis. Neither replaces the other.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can a background agent remediate an issue without a causal model?&lt;/strong&gt; It can work for specific events and runbooks that you have defined in advance, and it can work for localized, code-level problems. But in environments with many dependencies and complex topology, a background agent seeking to remediate an issue without a causal model is just as likely to make the situation worse as it is to fix the root cause. A causal model ensures the agent acts on the cause of the observed anomalies, rather than on the symptoms themselves.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is Open-Inspect safe to deploy across multiple teams?&lt;/strong&gt; Open-Inspect is single-tenant by design: everyone in a deployment shares the same GitHub App installation and repository scope. It's built for a trusted internal team with shared repository access, not for isolating multiple organizations or teams with different permissions within a single deployment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I trigger a background agent from a production alert?&lt;/strong&gt; Open-Inspect sessions can be triggered by webhooks, so a monitoring alert can start a session directly. The session then queries Causely for a diagnosis before taking any action, rather than starting from the raw alert payload. The causely-oss example documents this trigger pattern end-to-end.&lt;/p&gt;

</description>
      <category>agents</category>
      <category>sre</category>
    </item>
    <item>
      <title>Is access enough? Auth patterns for Claude Managed Agents</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 17 Aug 2026 19:52:38 +0000</pubDate>
      <link>https://dev.to/causely/is-access-enough-auth-patterns-for-claude-managed-agents-1ldg</link>
      <guid>https://dev.to/causely/is-access-enough-auth-patterns-for-claude-managed-agents-1ldg</guid>
      <description>&lt;p&gt;Originally posted to &lt;a href="https://dev.tourl"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/byemini/" rel="noopener noreferrer"&gt;Ben Yemini&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL; DR&amp;nbsp;
&lt;/h2&gt;

&lt;p&gt;Background agents run independently and can be triggered by events, making them a good fit for on-call work. Claude Managed Agents are one example, and MCP tunnels give them access to the environment&amp;nbsp;they're&amp;nbsp;making decisions over, reaching it, and authenticating. But access is just the starting point. An agent that can reach and log into every system still&amp;nbsp;isn't&amp;nbsp;automatically one&amp;nbsp;you'd&amp;nbsp;trust to run&amp;nbsp;on-call.&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do background agents fit on-call work?&amp;nbsp;
&lt;/h2&gt;

&lt;p&gt;Background agents differ from interactive assistants in one&amp;nbsp;key&amp;nbsp;way: they run on their own schedule, triggered by an event&amp;nbsp;(e.g., a page, an alert) rather&amp;nbsp;than by someone typing a prompt.&amp;nbsp;This&amp;nbsp;difference is what makes them&amp;nbsp;an ideal&amp;nbsp;fit for on-call&amp;nbsp;ops&amp;nbsp;work, where nobody&amp;nbsp;has to be&amp;nbsp;sitting at a keyboard when&amp;nbsp;performance degrades.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;Background agents persist. They pick up an event, run for as long as the investigation takes, from seconds to hours,&amp;nbsp;and can be triggered again later on the same thread.&amp;nbsp;&lt;a href="https://www.youtube.com/live/L0BXwymeYvk?t=216s" rel="noopener noreferrer"&gt;Cole&amp;nbsp;Murray,&amp;nbsp;the creator of&amp;nbsp;Open-Inspect,&amp;nbsp;walks&amp;nbsp;through this distinction in detail&lt;/a&gt;, and it's worth watching: the architecture decisions that make sense for a chat assistant don't automatically transfer to something that runs unattended.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;Claude Managed Agents&amp;nbsp;provide a similar instantiation:&amp;nbsp;their&amp;nbsp;sessions persist&amp;nbsp;in the&amp;nbsp;cloud, execute over time,&amp;nbsp;and can be triggered by automated events.&amp;nbsp; Anthropic's&amp;nbsp;own walkthrough of managed agents&amp;nbsp;covers the mechanics&amp;nbsp;and shows&amp;nbsp;&lt;a href="https://anthropic.ondemand.goldcast.io/on-demand/f8679328-af86-4d55-9f76-1f32dad49918" rel="noopener noreferrer"&gt;how to build an SRE&amp;nbsp;incident response agent&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;For on-call work, this approach is helpful in two ways. For reactive&amp;nbsp;workflows,&amp;nbsp;it ensures that whoever prompts the agent has the same setup and that the agent has the necessary access to assist with the investigation. More&amp;nbsp;importantly,&amp;nbsp;for proactive&amp;nbsp;workflows,&amp;nbsp;the agent can be triggered by events and start and end the investigation before an on-call engineer gets out of bed.&amp;nbsp;&amp;nbsp;&lt;/p&gt;

&lt;p&gt;Recently,&amp;nbsp;at Code with Claude in London on May 19, 2026, Anthropic shipped two features that&amp;nbsp;improve&amp;nbsp;the performance of these agents when running them against production systems.&amp;nbsp;Self-hosted sandboxes (public beta) and MCP tunnels (research preview). Together, they let a cloud-run agent reach a private Kubernetes cluster or an internal Prometheus instance without opening an inbound port, closing a gap that had been blocking some on-call use cases on security review&amp;nbsp;alone.&amp;nbsp;These&amp;nbsp;welcomed&amp;nbsp;improvements make this approach much more practical.&amp;nbsp;It also surfaces&amp;nbsp;the next key question: once the agent&amp;nbsp;has&amp;nbsp;access,&amp;nbsp;does it have the context it needs to resolve the problem?&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  Reaching your infrastructure is (mostly) a solved problem&amp;nbsp;
&lt;/h2&gt;

&lt;p&gt;A managed agent runs in&amp;nbsp;Anthropic's&amp;nbsp;cloud; your cluster, your metrics, and your logs run in your environment. The agent never touches them directly. It&amp;nbsp;calls MCP servers that do. Your cluster exposes a Kubernetes MCP&amp;nbsp;server,&amp;nbsp;your metrics stack exposes a Prometheus MCP server, and the agent calls&amp;nbsp;both as&amp;nbsp;tools. It never gets a raw&amp;nbsp;kubeconfig&amp;nbsp;or a database connection string.&amp;nbsp;&amp;nbsp;&lt;/p&gt;

&lt;p&gt;The first wall anyone hits&amp;nbsp;is&amp;nbsp;that a cloud agent&amp;nbsp;can't&amp;nbsp;reach&amp;nbsp;localhost&amp;nbsp;or a private API server. For local development or a&amp;nbsp;&lt;code&gt;kind&lt;/code&gt;&amp;nbsp;cluster, a tunnel like&amp;nbsp;cloudflared's&amp;nbsp;quick-tunnel mode gets you unblocked in minutes. This works for&amp;nbsp;a&amp;nbsp;proof&amp;nbsp;of concept, but&amp;nbsp;not for running production traffic. The production answer is&amp;nbsp;&lt;a href="https://platform.claude.com/docs/en/agents-and-tools/mcp-tunnels/overview" rel="noopener noreferrer"&gt;native MCP tunnels&lt;/a&gt;. A lightweight gateway inside your network opens a single outbound connection;&amp;nbsp;there's&amp;nbsp;no inbound port and no public MCP endpoint for a scanner to find. This is a general property of cloud-run agents, not something specific to Claude. Any agent platform running outside your perimeter&amp;nbsp;has to&amp;nbsp;solve the same reachability problem, and outbound-only tunneling is the shape the industry,&amp;nbsp;including&amp;nbsp;&lt;a href="https://aws.amazon.com/blogs/networking-and-content-delivery/network-connectivity-patterns-for-agents-deployed-on-amazon-bedrock-agentcore-runtime/" rel="noopener noreferrer"&gt;AWS&lt;/a&gt;&amp;nbsp;and&amp;nbsp;&lt;a href="https://docs.cloud.google.com/architecture/multi-agent-private-networking-patterns" rel="noopener noreferrer"&gt;Google&lt;/a&gt;,&amp;nbsp;has converged on this year.&lt;br&gt;
&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you authenticate an agent to systems it can now reach?
&lt;/h2&gt;

&lt;p&gt;&amp;nbsp;&lt;br&gt;
The Managed Agents MCP connector sends one thing to your server: an&amp;nbsp;&lt;code&gt;Authorization: Bearer&lt;/code&gt;&amp;nbsp;header. Credentials live in a vault matched by the MCP server's URL, not configured on the server entry itself. This&amp;nbsp;means the hard part of auth&amp;nbsp;isn't&amp;nbsp;the agent side,&amp;nbsp;it's&amp;nbsp;getting your existing auth&amp;nbsp;approach&amp;nbsp;to&amp;nbsp;work with&amp;nbsp;that&amp;nbsp;model.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8i6b68ssnyo06xpzb4kd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8i6b68ssnyo06xpzb4kd.png" alt="Three auth patterns for background-agents" width="800" height="560"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Three cases show up in practice, in&amp;nbsp;roughly escalating&amp;nbsp;difficulty:&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No auth.&lt;/strong&gt;&amp;nbsp;A Kubernetes MCP server behind a tunnel with cluster-level RBAC as the only gate. Fine for a&amp;nbsp;&lt;code&gt;kind&lt;/code&gt;&amp;nbsp;cluster or a low-stakes internal tool; not something to run against a production API server without at least a bearer token in front of it.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Static bearer token.&lt;/strong&gt;&amp;nbsp;A Grafana service-account token, stored in a vault and matched to the MCP server's URL. This is the common case for read-only observability queries.&amp;nbsp;Prometheus, Loki, and Grafana all support long-lived service-account tokens, and the vault handles rotation.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;OAuth client-credentials → JWT.&lt;/strong&gt;&amp;nbsp;The hard case, since most internal APIs&amp;nbsp;weren't&amp;nbsp;built expecting a bearer-only connector.&amp;nbsp;It sends&amp;nbsp;&lt;code&gt;Authorization: Bearer&lt;/code&gt;&amp;nbsp;and nothing else, so the vault has to complete the OAuth exchange up front and hand back a plain bearer token.&amp;nbsp;Sending the client secret as a separate field instead produces a&amp;nbsp;&lt;code&gt;400 unknown field&lt;/code&gt;&amp;nbsp;error, since the&amp;nbsp;connector's&amp;nbsp;schema only expects the one header.&amp;nbsp;For a straightforward approach to&amp;nbsp;handling&amp;nbsp;this see the repo at the end of this post.&amp;nbsp;&amp;nbsp;&lt;/p&gt;

&lt;p&gt;The generalizable rule holds regardless of which of the three cases&amp;nbsp;you're&amp;nbsp;in: the&amp;nbsp;connector's&amp;nbsp;interface is&amp;nbsp;&lt;code&gt;Authorization: Bearer&lt;/code&gt;. Anything more complex belongs in the vault, not in the agent's configuration.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;Once&amp;nbsp;that's&amp;nbsp;wired up&amp;nbsp;and authenticated, the reachability problem is&amp;nbsp;addressed, and the real work can begin.&amp;nbsp;&lt;br&gt;
&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  Access&amp;nbsp;isn't&amp;nbsp;understanding&amp;nbsp;
&lt;/h2&gt;

&lt;p&gt;An agent with full read access to Kubernetes, Prometheus, and your log pipeline still reasons&amp;nbsp;poorly about the system it is investigating.&amp;nbsp;Even the richest telemetry does not come with a model of what depends on what, or more specifically, what can cause what.&amp;nbsp;Metric series, multiple alerts firing, log lines, and even traces provide the agent with rich data to investigate and reason with, but the access to this data is not structured in a way that makes it effective.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;Give an agent&amp;nbsp;&lt;code&gt;kubectl&lt;/code&gt;, a&amp;nbsp;PromQL&amp;nbsp;endpoint, and a log query tool, and it can retrieve almost&amp;nbsp;anything&amp;nbsp;but retrieval&amp;nbsp;isn't&amp;nbsp;diagnosis.&amp;nbsp;This&amp;nbsp;is&amp;nbsp;not&amp;nbsp;a knock on the underlying telemetry.&amp;nbsp;&lt;a href="https://opentelemetry.io/docs/specs/semconv/" rel="noopener noreferrer"&gt;OpenTelemetry's semantic conventions&lt;/a&gt;&amp;nbsp;exist precisely to make spans, metrics, and logs consistent enough to reason&amp;nbsp;over&amp;nbsp;but consistent data&amp;nbsp;isn't&amp;nbsp;the same as a causal model of the system that produced it.&amp;nbsp;&amp;nbsp;&lt;/p&gt;

&lt;p&gt;We've&amp;nbsp;written before about&amp;nbsp;&lt;a href="https://www.causely.ai/blog/how-causal-reasoning-addresses-the-limitations-of-llms-in-observability" rel="noopener noreferrer"&gt;why LLMs alone hallucinate root causes&lt;/a&gt;&amp;nbsp;for the same underlying reason: pattern-matching over telemetry&amp;nbsp;isn't&amp;nbsp;the same as reasoning over a dependency graph. The distinction between monitoring,&amp;nbsp;knowing something is wrong,&amp;nbsp;and observability,&amp;nbsp;knowing why,&amp;nbsp;makes a related point from a different angle. An agent that can query telemetry has solved the first problem. Solving the second requires encoding dependency and causality structure somewhere: in a semantic layer, a service graph, or an explicit causal model, so the agent has something to reason against, not just something to query.&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&amp;nbsp;&lt;br&gt;
&lt;strong&gt;How do I connect a cloud AI agent to infrastructure it&amp;nbsp;can't&amp;nbsp;reach directly?&lt;/strong&gt;&amp;nbsp;Deploy an MCP server for the system you want the agent to reach, then bridge it with an outbound-only tunnel. For local development or a&amp;nbsp;kind&amp;nbsp;cluster, a quick tunnel like&amp;nbsp;cloudflared&amp;nbsp;works in minutes. For production, use native MCP tunnels: a gateway inside your network opens a single outbound connection, so no inbound port or public MCP endpoint is ever exposed.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does giving an agent more system access make it more reliable?&lt;/strong&gt;&amp;nbsp;Not by itself. Raw access to Kubernetes, Prometheus, and logs lets an agent retrieve data, but that doesn't mean it can diagnose. Without an encoded model of service dependencies and event ordering, an agent&amp;nbsp;can't&amp;nbsp;reliably tell a root cause from a downstream symptom. It can query more, not reason better. Access solves reachability; understanding requires a separate structural layer.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's&amp;nbsp;the difference between an agent reaching a system and an agent understanding it?&lt;/strong&gt;&amp;nbsp;Reaching a system means the agent can call an MCP server and retrieve metrics, logs, or cluster state. Understanding means the agent can tell which of those signals is the&amp;nbsp;cause&amp;nbsp;and which are downstream effects,&amp;nbsp;something that requires a dependency graph or causal model, not just a wider set of tools to query.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I authenticate a managed agent to an internal API that uses OAuth?&lt;/strong&gt;&amp;nbsp;Resolve the OAuth client-credentials flow inside your credential vault, not in the agent's configuration. The Managed Agents MCP connector only ever sends an&amp;nbsp;&lt;code&gt;Authorization: Bearer&lt;/code&gt;&amp;nbsp;header matched to the server's URL.&amp;nbsp;It&amp;nbsp;won't&amp;nbsp;perform a token exchange or accept additional OAuth fields, so the vault has to mint the JWT and return a plain bearer token.&amp;nbsp;See the reference repo below for code examples.&amp;nbsp;&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do next&amp;nbsp;
&lt;/h2&gt;

&lt;p&gt;Read the&amp;nbsp;&lt;a href="https://github.com/causely-oss/background-agents/tree/main/claude-managed-agents" rel="noopener noreferrer"&gt;three-auth-pattern, kind-runnable reference repo&lt;/a&gt;&amp;nbsp;to see all three cases running end to end&amp;nbsp;with a Claude managed SRE agent accessing&amp;nbsp;a real cluster. The open question this setup raises&amp;nbsp;is&amp;nbsp;whether adding a causal or semantic layer on top of raw telemetry access measurably changes how well an agent investigates an incident.&amp;nbsp;We're&amp;nbsp;running a reproducible, measured comparison on&amp;nbsp;this&amp;nbsp;next. Stay tuned.&amp;nbsp;&amp;nbsp;&lt;/p&gt;

</description>
      <category>mcp</category>
      <category>backgroundagents</category>
      <category>ai</category>
      <category>sre</category>
    </item>
    <item>
      <title>Semantics in Observability: A Precise Vocabulary for System Understanding</title>
      <dc:creator>Ben Yemini</dc:creator>
      <pubDate>Mon, 03 Aug 2026 18:32:30 +0000</pubDate>
      <link>https://dev.to/causely/semantics-in-observability-a-precise-vocabulary-for-system-understanding-2fdp</link>
      <guid>https://dev.to/causely/semantics-in-observability-a-precise-vocabulary-for-system-understanding-2fdp</guid>
      <description>&lt;p&gt;Orignially posted to &lt;a href="https://www.causely.ai/blog/semantics-in-observability" rel="noopener noreferrer"&gt;causely.ai&lt;/a&gt; by &lt;a href="https://www.linkedin.com/in/endresara" rel="noopener noreferrer"&gt;Endre Sara&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The word &lt;strong&gt;context&lt;/strong&gt; is overloaded in observability discourse to the point of losing precision. It is used to refer to shared labels, a topology map, temporal proximity, and propagated trace identifiers. Four distinct things, each useful, none of them sufficient.&lt;/p&gt;

&lt;p&gt;This blog defines a precise, layered model of observability semantics intended for engineers and SREs building or operating complex distributed systems.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Gap in Current Observability Tooling
&lt;/h2&gt;

&lt;p&gt;Most observability tooling today implements a shallow semantics of entity and some form of topology, often referred to as the knowledge graph – typically derived from trace data – with partial reach into behavior through threshold-based alerting and anomaly detection. A knowledge graph is a generic term that does not define the semantics of what it captures. It is typically assumed to be a list of services and their topological relationships, but as we point out later, this is not sufficient to answer questions. It is also a dynamic, ever-changing graph; capturing it only once, or once a week, is not a reliable representation for real-time, continuous, automated operation. Deeper semantics are largely absent from tooling and instead exist as undocumented knowledge held by individual engineers.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;This lack of semantics has predictable operational consequences:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Alert fatigue: alerting systems fire on symptoms without the ability to identify the root cause, producing storms where a single fault generates dozens of notifications&lt;/li&gt;
&lt;li&gt;Slow incident response: root cause reasoning and impact analysis happen manually, in real time, under pressure, because the causal and dependency knowledge exists only in human memory&lt;/li&gt;
&lt;li&gt;Blast-radius estimation is guesswork: a lack of knowledge of the impact of failure. There is no systematic way to ask what would happen if a given component failed, which dependent services would be affected, which symptoms would surface, and in what order.&lt;/li&gt;
&lt;li&gt;Causal knowledge is reactive, not preventive: no proactive reasoning about causes. Engineers cannot ask what classes of failure could produce a given set of symptoms, or which components in the current system state are most likely to trigger an incident.&lt;/li&gt;
&lt;li&gt;No counterfactual reasoning: there is no way to assert that a specific condition could cause a specific failure, or conversely that a given failure mode cannot produce a set of observed symptoms. Without formal causal and dependency knowledge, both directions of reasoning – “could this cause that?” and “could that have caused this?” – rely entirely on individual experience&lt;/li&gt;
&lt;li&gt;Knowledge loss: engineers who have internalized the causal and constraint knowledge leave, and the organization reverts to slower, less reliable incident response&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The data to address these problems is largely available in service meshes, infrastructure APIs, configuration management systems, and telemetry pipelines. The gap is not data. It is the absence of explicit semantic models that make that data legible to AI Agents. The sections below define the models.&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  A Layered Semantic Model for Observability
&lt;/h2&gt;

&lt;p&gt;The following six layers describe the semantic structure required to move from “we see a signal” to “we understand what is happening and why.” Each layer builds on the previous. Tooling that skips layers will exhibit predictable gaps in reasoning capability.&lt;/p&gt;

&lt;p&gt;It is important to note that the semantics model at each layer has two parts, a knowledge base and a dynamic graph. The knowledge base captures the generic knowledge of what may be observed. The knowledge base captures abstract semantic descriptions common across all environments. The dynamic graph instantiates the knowledge base to generate a real-time model of the managed environment at a given point in time. The dynamic graph is the actual observed reality, specific to a given environment at a given point in time, and keeps adapting as the environment changes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 1: Entity Model
&lt;/h2&gt;

&lt;p&gt;The &lt;strong&gt;entity model&lt;/strong&gt; defines the named objects in the system and the attributes that describe them. An entity is any discrete thing with identity and an observable state.&lt;/p&gt;

&lt;p&gt;The entity knowledge base captures the types of entities that may be discovered, e.g., a service instance, a host, a Kubernetes pod, a database cluster, a message queue, a CDN edge node, an AI Agent, an MCP server with tools or an LLM model and its provider, and the attributes that may be observed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Identity&lt;/strong&gt;: UUID with associated attributes like name, namespace, cluster, region, environment, labels&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Configuration&lt;/strong&gt;: resource limits, replica count, connection pool size, JVM flags&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runtime state&lt;/strong&gt;: CPU utilization, memory used, request rate, error rate, queue depth&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The dynamic entity model instantiates the knowledge base with the discovered entities and serves as an inventory of all entities in the environment at a given point in time. For example, the list of services, hosts, pods, databases, message queues, etc., in the environment.&lt;/p&gt;

&lt;p&gt;Without an entity model, telemetry is a stream of dimensioned numbers. With one, it is a description of a system composed of named, typed, queryable objects. This semantics is a prerequisite for all higher-order reasoning.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 2: Topology Model
&lt;/h2&gt;

&lt;p&gt;The topology model encodes the structural relationships between entities: which services call which databases, which pods run on which nodes, which queues are consumed by which workers, etc.&lt;br&gt;
The topology knowledge base captures the relationship that may exist between types of entities, For exaple, a service may be connected to another service, a service may be accessing a database, a service may be layered over a pod.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbqg1m1fp48nwfsd0rkxi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbqg1m1fp48nwfsd0rkxi.png" alt="Relationship Types" width="799" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Relationship Types&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The dynamic topology graph represents the actual discovered relationships between discovered entities. For example, the Coordinator Agent is connected to the Research, Coding, and Ops subagents; the GitHub MCP server, which includes the create pull request tool; and the GPT-4o model, which is layered over the OpenAI model provider (see Figure 1).&lt;/p&gt;

&lt;p&gt;It is important to note that the knowledge base captures design-time assertions, not runtime observations. This distinction is critical. A service dependency map derived from trace data reflects what happened during the observation window. The dynamic topology graph reflects what is. Traces may be incomplete (non-instrumented paths, async operations, batch jobs). The dynamic topology graph enables blast-radius analysis; given that entity foo is degraded, which other entities are structurally dependent on it and therefore at risk.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fma08trorwzgly5zibucj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fma08trorwzgly5zibucj.png" alt="Figure 1: Topology — structural relationships between agents, MCP servers, and Model Providers." width="800" height="546"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Figure 1: Topology — structural relationships between agents, MCP servers, and Model Providers.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 3: Behavior Model
&lt;/h2&gt;

&lt;p&gt;A &lt;strong&gt;behavior model&lt;/strong&gt; captures the observable signals of normal and abnormal system behavior. It defines &lt;strong&gt;symptoms&lt;/strong&gt; – semantically meaningful deviations from expected state  – and &lt;strong&gt;events&lt;/strong&gt; – discrete state transitions that represent changes in entity health or configuration.&lt;/p&gt;

&lt;p&gt;A raw metric is not a symptom. The number 95.3 on a CPU utilization gauge is not meaningful without knowing the entity it describes, the threshold at which that utilization constitutes an anomaly, and the operational context in which that threshold applies. A symptom is the semantic interpretation: &lt;em&gt;this entity is exhibiting CPU saturation relative to its configured capacity and expected workload.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The behavior model is what converts monitoring data into an actionable signal. Alerting systems that operate directly on raw metrics without a behavior model produce high false-positive rates because they lack the semantic layer needed to distinguish signal from noise.&lt;/p&gt;

&lt;p&gt;The behavior knowledge base captures the possible observable anomalies an entity may experience. For example, a service may be degraded, experiencing high latency, or a node may be highly utilized, experiencing high CPU utilization.&lt;/p&gt;

&lt;p&gt;The dynamic behavior model represents the environment's actual runtime state based on what is being observed. For example, the Research subagent is degraded and experiencing high latency, or the Postgres MCP is congested and experiencing high query times.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 4: Causality Model
&lt;/h2&gt;

&lt;p&gt;The &lt;strong&gt;causality model&lt;/strong&gt; captures the cause-and-effect relationship between potential causes and the symptoms or events they may cause.&lt;br&gt;
The causal knowledge base describes the potential causes, the symptoms they may cause, and how they may propagate. Propagation captures how a cause occurring on one entity may propagate to another related entity. &amp;nbsp;&lt;/p&gt;

&lt;p&gt;The causal knowledge base captures generic knowledge about the types of causes that may occur in an environment and how they will manifest, i.e., the symptoms that may be observed when they occur.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An MCP service may be congested and when it is congested it may cause its tool calls to be slow, manifested as a high tool-call latency symptom, and may propagate and cause degradation of subagents that depend on it.&lt;/li&gt;
&lt;li&gt;A subagent may be degraded and when it is degraded it may cause its responses to be slow, manifested as a slow response time symptom, and may propagate and cause degradation of the coordinator agent that orchestrates it.&lt;/li&gt;
&lt;li&gt;A model provider may be degraded and when it is degraded it may cause its inference requests to be slow, manifested as high inference latency, and may propagate to subagents using its models causing them to be degraded.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As the above examples illustrate, the causal knowledge base is completely independent of a given environment. It is independent of how many providers, MCP services, and/or subagents are in the environment, and which models run on which providers, which subagents access which MCPs, or which coordinator orchestrates which subagents.&lt;/p&gt;

&lt;p&gt;The dynamic causal graph represents the actual runtime causality in the environment at a given point in time. It is a function of the dynamic topology graph and represents all potential specific causes that may occur in the environment, along with the specific symptoms they may cause. The figure below illustrates a concrete example: the Postgres MCP may be congested, and the fault may propagate through the execute_sql tool call to the Coding and Ops subagents, and then onward to the Coordinator, with symptom probability decreasing at each hop. Similarly, Anthropic provider degradation may propagate through Claude inference to the Research, Coding, and Ops subagents, and then to the Coordinator.&lt;/p&gt;

&lt;p&gt;In this example, the dynamic causal graph captures that a congested Postgres MCP may cause degradation of the Coding and Ops subagents and the Coordinator. It is important to note that the keyword semantically is “may”. A congested Postgres MCP may degrade downstream entities, but that doesn’t necessarily mean the Postgres MCP is congested – Anthropic provider degradation could produce overlapping symptoms. To conclude, the cause of an observed symptom requires analyzing the state of all potential symptoms.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fecwu3z8sxui6krk4flkx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fecwu3z8sxui6krk4flkx.png" alt="Causal graph derived from the agentic topology in Figure 1. Two example fault paths (Postgres MCP congestion, Anthropic provider degradation) propagating through MCP services and subagents into the coordinator." width="799" height="247"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Figure 2: Causal graph derived from the agentic topology in Figure 1. Two example fault paths (Postgres MCP congestion, Anthropic provider degradation) propagating through MCP services and subagents into the coordinator.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Cause (left) propagates rightward through topology edges. Probabilities indicate the likelihood of each symptom given the cause. Multiple symptoms may share a cause; diagnosis evaluates the full set of observed symptoms against this graph.&lt;/p&gt;

&lt;p&gt;The causality model is what makes root cause analysis tractable. Without it, a single infrastructure fault that produces symptoms across dozens of dependent services generates an alert storm with no structural differentiation between cause and effect. With it, symptoms can be suppressed or annotated as consequences once their cause is pinpointed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 5: Attributes Dependency Model
&lt;/h2&gt;

&lt;p&gt;Where the causality model operates at the symptom level, the &lt;strong&gt;attributes dependency model&lt;/strong&gt; operates at the attribute level. As its name suggests, it captures attribute dependencies. The dependent attributes can be of the same entity or of different entities. Dependency knowledge captures that attributes are correlated, along with the directionality and function of the correlation. The function can be defined in the model or learned.&lt;/p&gt;

&lt;p&gt;The attribute knowledge base captures the generic knowledge of which attribute depends on which. Examples of such dependencies:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Subagent response latency is a function of the tool-call duration of the MCP services it invokes&lt;/li&gt;
&lt;li&gt;Coordinator response latency is a function of the response latencies of the subagents it orchestrates&lt;/li&gt;
&lt;li&gt;Subagent inference latency is a function of the inference latency of the model provider serving its model&lt;/li&gt;
&lt;li&gt;A subagent's token throughput is a function of the rate at which the coordinator dispatches requests to it.&lt;/li&gt;
&lt;li&gt;The end-to-end coordinator latency is a function of both per-call latencies and the number of subagent calls per task. A non-linear relationship, since a coordinator may fan out to multiple subagents in parallel, retry on failure, or recurse on intermediate results.&lt;/li&gt;
&lt;li&gt;Total tokens consumed by a subagent are a function of both the per-call token cost and the number of model-provider inference calls, which scales non-linearly with conversation depth, tool-call retries, and self-reflection loops.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The dynamic attribute dependency graph represents the actual run-time dependencies in the environment at a given point in time. It is a function of the dynamic topology model. Given the above static model examples, the dynamic model examples would be:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If the Coding subagent is accessing the Postgres MCP, Coding subagent latency is a function of execute_sql tool-call duration&lt;/li&gt;
&lt;li&gt;If the Coordinator is orchestrating the Coding subagent, Coordinator latency is a function of Coding subagent latency. Note that based on this, Coordinator latency can be computed as a function of execute_sql tool-call duration, even though the Coordinator does not directly access the Postgres MCP.&lt;/li&gt;
&lt;li&gt;If the Research subagent is using Claude served by Anthropic, the Research subagent's inference latency is a function of Anthropic's Claude inference latency.&lt;/li&gt;
&lt;li&gt;For the Ops subagent, its token throughput is a function of the rate of requests the Coordinator dispatches to it.&lt;/li&gt;
&lt;li&gt;If the Coordinator fans out a single user task to the Research, Coding, and Ops subagents in parallel, total Coordinator latency is approximately the max of the three subagent latencies, but if any subagent retries on tool-call failure, that subagent's effective latency multiplies by its retry count, so end-to-end latency grows non-linearly with failure rate.&lt;/li&gt;
&lt;li&gt;If the Coding subagent calls the Postgres MCP once per row in a result set, Coding subagent latency is a function of execute_sql duration multiplied by row count, so a query returning N rows produces N tool calls, and end-to-end latency scales with N rather than staying constant.&lt;/li&gt;
&lt;li&gt;If the Research subagent invokes Claude in a multi-turn reasoning loop with branching factor B and depth D, total Anthropic inference calls scale as B^D – so doubling reasoning depth roughly squares the inference cost, even though the Research subagent's per-task topology entry is unchanged.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhv332lnulrh3haibzh4m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhv332lnulrh3haibzh4m.png" alt="Non-linear call-count amplification. Three runtime patterns where total cost diverges from the static topology in Figure 1. (A) Parallel fan-out with retries multiplies the worst-case branch latency. (B) Per-row tool-call loops produce linear-in-N call counts that the topology graph cannot bound. (C) Recursive reasoning produces B^D inference calls, exponential in depth." width="800" height="350"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Figure 3: Non-linear call-count amplification. Three runtime patterns where total cost diverges from the static topology in Figure 1. (A) Parallel fan-out with retries multiplies the worst-case branch latency. (B) Per-row tool-call loops produce linear-in-N call counts that the topology graph cannot bound. (C) Recursive reasoning produces B^D inference calls, exponential in depth.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The attribute dependency graph enables change impact analysis: before modifying a configuration parameter, an operator or automation system can trace the dependency graph to identify which runtime attributes will be affected, on which entities, with what expected magnitude.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 6: Constraint Model
&lt;/h2&gt;

&lt;p&gt;The &lt;strong&gt;constraint model&lt;/strong&gt; defines the desired state. The desired state is one in which goals are achieved while constraints are satisfied. Each entity in an environment may have a desired state, but, more interestingly, the environment as a whole may have one as well.&lt;/p&gt;

&lt;p&gt;A desired environment state is one in which all applications deliver on their goals while running within their operational constraints. Usually, the goals are performance goals of latency and throughput, and service level goals in terms of errors and availability, while the operational constraints are capacity, budget, compliance, and configuration.&lt;/p&gt;

&lt;p&gt;An entity's desired state is the configuration space within which the entity and its attributes operate without triggering symptoms or degradation events. It is the semantic encoding of the system’s operational envelope.&lt;/p&gt;

&lt;p&gt;The constraint model captures both the environment's desired state and the entity's desired state. These can be defined independently, but by using the attribute dependency model, the entity's desired state can be generated/inferred from the environment's desired state, resulting in a very compelling and powerful system.&lt;/p&gt;

&lt;p&gt;As in all other layers, the constraint knowledge base captures generic knowledge about constraints, and the dynamic constraint model represents the runtime constraints at a given point in time based on the discovered topology. Constraints are not just thresholds on metrics. They are relationships between configuration parameters and the runtime conditions required for healthy operation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Headroom constraint: Container memory limit &amp;gt;= 1.4 x used heap&lt;/li&gt;
&lt;li&gt;Budget constraint: Application cost &amp;lt; budget&lt;/li&gt;
&lt;li&gt;Latency constraint: Service latency &amp;lt; 100ms&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When the constraint model is explicit, it becomes possible to answer operational questions that today require senior engineering intuition: “What is the minimum safe memory limit for this service at current traffic?” “If we reduce the connection pool by 40%, at what request rate will we begin to see queuing symptoms?” The constraint model is the missing link between observability (what is happening) and operations (what configuration produces health). It is the semantic layer connecting configuration management to runtime behavior.&amp;nbsp;&lt;/p&gt;

&lt;h2&gt;
  
  
  What “Context” Actually Covers and Where It Falls Short
&lt;/h2&gt;

&lt;p&gt;Most observability platforms use the word loosely. Here is what the four common implementations actually encode, and where each one stops.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Shared Labels&lt;/strong&gt;&lt;br&gt;
Common attributes – &lt;em&gt;service, environment, region, version&lt;/em&gt; – are attached to metrics, logs, and traces. This allows tooling to pivot between signal types for a given entity. The relationship is implied by label equality, not by any explicit model of dependency or causation. Two data points sharing &lt;em&gt;service=checkout&lt;/em&gt; are navigable together; they are not necessarily related by anything other than that label. This is a partial implementation of the entity model (layer 1) that lacks declarative topological relationships and behavior.&amp;nbsp;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Topology Map&lt;/strong&gt;&lt;br&gt;
A topology map encodes structural relationships between entities, which services call which databases, which pods run on which nodes, and which workloads share a cluster. It is the most semantically substantive form of “context” in common tooling and is typically derived from trace data, service-mesh telemetry, or infrastructure APIs. This is a partial implementation of the topology model (layer 2): it captures that entity X is connected to entity Y, but not what behavior propagates along that edge, why a fault on X causes a symptom on Y, or which of X’s attributes drives which of Y’s. A topology map shows the wiring; it does not encode behavior (layer 3), causality (layer 4), attribute dependency (layer 5), or constraints (layer 6). Two services connected in the topology may be tightly coupled in failure propagation or entirely decoupled. The map alone cannot distinguish them. In practice, this means a topology map supports navigation (“show me what this service talks to”) but not reasoning (“if this service degrades, what symptoms should I expect, on which dependents, and why”). That reasoning requires the higher layers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Temporal Proximity&amp;nbsp;&lt;/strong&gt;&lt;br&gt;
Events co-located on a timeline are assumed by operators to be potentially related. The system carries no model of that relationship; pattern recognition is entirely delegated to the human. This is correlation by coincidence, not by structure. It addresses none of the six semantic layers explicitly; it is a UI affordance for human reasoning, not a semantic model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Distributed Trace Propagation&lt;/strong&gt;&lt;br&gt;
A trace context header (W3C traceparent, B3, etc.) propagated across service boundaries encodes a genuine causal chain: span B was initiated by span A. This is the most semantically rich form of “context” in common use. It partially implements the topology model (layer 2) for instrumented call paths, and provides raw material for the behavior model (layer 3). However, it is constrained to synchronous, instrumented code paths and does not capture async operations, batch jobs, or infrastructure-level dependencies. It carries no causality, dependency, or constraint models.&lt;/p&gt;

&lt;p&gt;In summary, shared labels, temporal proximity, topology, and trace propagation collectively reach into layers 1 and 2. Layers 4 through 6 – causality, dependency, and constraints – are entirely outside the scope of what “context” addresses. That is the gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion&amp;nbsp;
&lt;/h2&gt;

&lt;p&gt;Context – shared labels, topology map, temporal proximity, trace propagation – is a necessary foundation for navigating observability data. It is not sufficient for understanding systems. The jump from “we can see the signals” to “we understand what is happening and why” requires explicit semantic models at six distinct layers: entity, topology, behavior, causality, dependency, and constraint.&lt;/p&gt;

&lt;p&gt;The practical implication for SREs and platform engineers is that runbooks, tribal knowledge, and escalation paths are informal implementations of layers 4 through 6. The question is not whether to encode this knowledge, which already exists, but whether to leave it implicit in human memory or make it explicit in a form that tooling can reason about.&lt;br&gt;
Making semantics explicit is what makes a system legible to an AI agent. An agent reasoning over shared labels, topology, and temporal proximity alone is restricted to the same navigation problem a human faces, pivoting between signals without a model of why they are related. An agent reasoning over layers 4 through 6 can answer the questions that actually drive incident response, capacity planning, and change approval: what is causing this symptom, what is the blast radius of a degraded component, what configuration change would restore the constraint, and what would happen if that change were made. These questions do not require new data; the telemetry, configuration, and trace propagation already carry the raw signal. What they require is a model that the agent can reason against. One that encodes the causal, dependency, and constraint knowledge that today lives only in the engineers who happen to remember.&lt;/p&gt;

&lt;p&gt;Semantics is not an observability feature. It is the foundation that makes observability useful.&lt;/p&gt;

</description>
      <category>semantics</category>
      <category>context</category>
      <category>causality</category>
    </item>
  </channel>
</rss>
