<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Bonree-Js</title>
    <description>The latest articles on DEV Community by Bonree-Js (@bonree-js).</description>
    <link>https://dev.to/bonree-js</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4124005%2Fb0212323-ae4e-4db3-842c-d1cbf4d57cab.jpg</url>
      <title>DEV Community: Bonree-Js</title>
      <link>https://dev.to/bonree-js</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/bonree-js"/>
    <language>en</language>
    <item>
      <title>Enterprise AI Observability: Choosing an AI Observability Platform That AI, SRE and Security Teams Can All Use</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Fri, 13 Nov 2026 02:31:48 +0000</pubDate>
      <link>https://dev.to/bonree-js/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai-observability-4hn6</link>
      <guid>https://dev.to/bonree-js/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai-observability-4hn6</guid>
      <description>&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Enterprise AI Observability: Choosing an AI Observability Platform That AI, SRE and Security Teams Can All Use"&lt;/span&gt;
&lt;span class="na"&gt;published&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"A selection guide for enterprise AI observability platforms, built around what AI, SRE and security teams each need and how to test candidates against a real scenario."&lt;/span&gt;
&lt;span class="na"&gt;tags&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;observability, ai, devops, platformengineering&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An AI observability platform is the system that lets a team see how its AI applications behave in production, including model calls, retrieval, agent steps, cost and the services underneath them. In an enterprise, the hard part of choosing one is rarely the feature list. It is that several teams will depend on the answer, and each of them is asking a different question.&lt;/p&gt;

&lt;p&gt;Imagine a selection review with three people in the room: an engineer from the AI team, a site reliability engineer, and someone from security. The discussion tends to go in circles unless the group agrees on what each person needs to be able to do afterwards.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with what each person must be able to do
&lt;/h2&gt;

&lt;p&gt;The AI engineer needs to answer: which model call or tool step caused this response to be wrong, and did it get worse after the last change? That requires visibility into prompts, responses, retrieval and agent steps, plus a way to compare versions.&lt;/p&gt;

&lt;p&gt;The SRE needs to answer: why is this slow or failing, and which team owns the fix? That requires the AI application to be connected to the services and infrastructure it calls, along with alerting and fault localization that work across both.&lt;/p&gt;

&lt;p&gt;The security reviewer needs to answer: what sensitive data did we collect, how was it protected, and who touched it? That requires masking, access control and an audit trail of which identity called which tool, along with deployment options that match data requirements.&lt;/p&gt;

&lt;p&gt;If a candidate platform lets only one of these three people complete their task, the decision is not finished.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four screening questions
&lt;/h2&gt;

&lt;p&gt;Before any demo, these four questions filter most candidates:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Does it cover the kinds of AI applications we run, both LLM applications and agents?&lt;/li&gt;
&lt;li&gt;Can it follow one request from the application through model calls and tool calls?&lt;/li&gt;
&lt;li&gt;Does it connect AI applications with the underlying IT environment instead of treating them separately?&lt;/li&gt;
&lt;li&gt;Does it support analysis and response, such as anomaly analysis, fault localization and assistance with operations?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Test with a scenario, not a spec sheet
&lt;/h2&gt;

&lt;p&gt;Pick a realistic scenario, such as an AI feature that slows down after a release, and run it past all three people on each candidate. The question is simple: can each of them reach their own answer from the same trace? A platform that satisfies the AI team but sends the SRE to a second tool has moved the problem rather than solved it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the shared record matters
&lt;/h2&gt;

&lt;p&gt;Different tools produce different partial truths. When AI, operations and security each use a different view, a disagreement about the cause of an incident is often really a disagreement about which data to trust. A shared trace turns that into a discussion about the same evidence. It is also what agentic AI operations depends on: agents that help with diagnosis can only be trusted if they can point to telemetry that people can verify.&lt;/p&gt;

&lt;h2&gt;
  
  
  Warning signs during evaluation
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;AI telemetry lives apart from application and infrastructure telemetry&lt;/li&gt;
&lt;li&gt;Instrumentation cannot be exported or standardized, which makes later migration costly&lt;/li&gt;
&lt;li&gt;Prompts and responses are stored without configurable masking&lt;/li&gt;
&lt;li&gt;Sampling drops the failures you most need to investigate&lt;/li&gt;
&lt;li&gt;Data volume costs cannot be predicted as usage grows&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some vendors now package AI observability with application and infrastructure monitoring in a single platform. &lt;a href="https://en.bonree.com" rel="noopener noreferrer"&gt;Bonree ONE&lt;/a&gt; is one example of that approach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Quick answers
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Can a general observability tool cover AI applications?&lt;/strong&gt;&lt;br&gt;
Partly. It covers services and infrastructure, but model calls, prompts, retrieval, agent steps and token cost need to be captured and tied back to the rest of the stack.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who should make the decision?&lt;/strong&gt;&lt;br&gt;
A small group with AI, operations and security representation, so that the platform works for everyone who will depend on it.&lt;/p&gt;




&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw7qeshzd4q3s9vmtle1x.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw7qeshzd4q3s9vmtle1x.png" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>llm</category>
      <category>agents</category>
    </item>
    <item>
      <title>I Agent Observability in Production: 5 Failure Patterns and How Agentic AI Operations Responds</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Thu, 12 Nov 2026 04:03:17 +0000</pubDate>
      <link>https://dev.to/bonree-js/i-agent-observability-in-production-5-failure-patterns-and-how-agentic-ai-operations-responds-2of1</link>
      <guid>https://dev.to/bonree-js/i-agent-observability-in-production-5-failure-patterns-and-how-agentic-ai-operations-responds-2of1</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpafdudyzvw97u4gd99mj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpafdudyzvw97u4gd99mj.png" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;---
title: "AI Agent Observability in Production: 5 Failure Patterns and How Agentic AI Operations Responds"
published: right
description: "A practical diagnosis path for AI agent failures in production: five recurring patterns, what telemetry reveals them, and how operations teams respond."
tags: observability, ai, devops, opentelemetry

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;AI agent observability means being able to explain, from telemetry alone, why an agent behaved the way it did in production: which decisions it made, which tools it called, and where time and money went. Without it, agentic AI operations become guesswork, because there is nothing to diagnose from.&lt;/p&gt;

&lt;p&gt;Most teams discover this after launch. The prototype worked in a demo, the dashboards showed healthy services, and yet users complained. This article lays out a diagnosis path, from the run to the step to the dependency, and uses it to walk through five patterns that account for many production agent failures.&lt;/p&gt;

&lt;h2&gt;
  
  
  The diagnosis path: run, step, dependency
&lt;/h2&gt;

&lt;p&gt;When an agent misbehaves, start from the whole run and ask whether it was unusual: too long, too slow, too costly, or ending in an answer that does not match its evidence. Then move to the individual step where the run went wrong. Finally, look at the dependency behind that step, such as a database, an API or an inference service. The five patterns below are easiest to recognize at one of these three levels.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 1: the answer that sounds right and is not
&lt;/h2&gt;

&lt;p&gt;Consider an agent that looks up an account, receives an empty result because of a permissions error, and then answers the user as though the account had no activity. Every request-level metric is green.&lt;/p&gt;

&lt;p&gt;This is diagnosed at the step level. Record whether each tool call actually succeeded, independently of whether the call was delivered. Then look for runs in which a final answer comes after a failed or empty step. Treat that combination as a quality incident, not a transport error.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 2: the run that never quite ends
&lt;/h2&gt;

&lt;p&gt;Suppose a tool keeps returning an unhelpful message and the agent keeps asking again with slightly different wording. Each call is fast and valid. Together they are expensive.&lt;/p&gt;

&lt;p&gt;This shows up at the run level: unusually long runs, repeated calls with similar arguments, and cost per run drifting upward. Step budgets and alerts on outlier run length turn an invisible loop into something an on-call engineer can act on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 3: slow, but nobody's fault
&lt;/h2&gt;

&lt;p&gt;An agent takes twelve seconds to respond. The model call took a few seconds, the retrieval took a few more, and a downstream service added the rest. No single piece breaches its own threshold.&lt;/p&gt;

&lt;p&gt;The remedy is to decompose each run into model time, tool time and dependency time, and to find the stage on the critical path. This is also where agent telemetry has to meet infrastructure telemetry: the slow dependency is often a conventional service, and it can only be named if both views share a trace.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 4: the bill grows faster than traffic
&lt;/h2&gt;

&lt;p&gt;Context accumulates across steps, tool results are passed back in full, and the same lookups are repeated. Spend rises while usage looks flat.&lt;/p&gt;

&lt;p&gt;Look at input size per step and the size of tool results, grouped by agent, tool and caller. The largest tool outputs are frequently the cheapest thing to trim.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pattern 5: the failure nobody can reproduce
&lt;/h2&gt;

&lt;p&gt;A user reports a failure. An engineer replays the same input and gets a correct answer. Model output varies, tool data changes, and nobody recorded which model, prompt and tool versions were in play.&lt;/p&gt;

&lt;p&gt;The fix is to record enough per step to reconstruct the decision path, and to attach version information to every run. Sensitive prompts and results should be masked according to policy before they are stored.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turning diagnosis into operations
&lt;/h2&gt;

&lt;p&gt;Finding a pattern once is useful. Operating at scale means making the response repeatable: alert on patterns instead of raw counters, route each incident to the team that owns the failing stage, and correlate agent behavior with the services underneath it.&lt;/p&gt;

&lt;p&gt;This is also where agents themselves start to help. An operations agent that assists with diagnosis should be held to the same standard as the system it examines: it should point to telemetry as evidence, and its actions should be observable and approved. Teams using platforms like &lt;a href="https://en.bonree.com" rel="noopener noreferrer"&gt;Bonree ONE&lt;/a&gt; can examine agent runs alongside application and infrastructure monitoring in one place, and its Sage AI (an AI Ops Agent Workbench) gives them a natural-language way to run operational Q&amp;amp;A, analysis and diagnosis with agents on top of that data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist before you ship
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One trace per run, with a span for each model call and tool call&lt;/li&gt;
&lt;li&gt;Tool outcome recorded separately from delivery status&lt;/li&gt;
&lt;li&gt;Step count, token use and latency tracked per run&lt;/li&gt;
&lt;li&gt;Model, prompt and tool versions attached to each run&lt;/li&gt;
&lt;li&gt;Masking in place before telemetry is stored&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Quick answers
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What should I monitor first for an AI agent in production?&lt;/strong&gt;&lt;br&gt;
Start with tool outcomes per step, run length, latency by stage and cost per run. Those four reveal most of the patterns above.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is agentic AI operations the same as AIOps?&lt;/strong&gt;&lt;br&gt;
They overlap. Agentic AI operations emphasizes running agents safely in production, and using agents to assist operations work, with telemetry as evidence.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>agents</category>
      <category>llm</category>
    </item>
    <item>
      <title>Building Trustworthy Agentic AI Operations: A Practical Roadmap from Read-Only Agents to Bounded Autonomy</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Wed, 04 Nov 2026 03:51:33 +0000</pubDate>
      <link>https://dev.to/bonree-js/building-trustworthy-agentic-ai-operations-a-practical-roadmap-from-read-only-agents-to-bounded-55d9</link>
      <guid>https://dev.to/bonree-js/building-trustworthy-agentic-ai-operations-a-practical-roadmap-from-read-only-agents-to-bounded-55d9</guid>
      <description>&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;


AI agents are beginning to take on real IT operations work: inspecting services, correlating alerts, investigating incidents, querying tools and preparing remediation actions.

But the technical ability to call an API or execute a command is not the same as being ready to do so in production.

For an operations team, the difficult question is not simply whether an agent can perform a task. It is **what authority should it have, what evidence is required, and when is it safe to expand its scope?**

Trustworthy agentic operations require more than capable models. Teams need visibility into execution, enforceable permission boundaries, verifiable outcomes and clear accountability.

A practical adoption model should start with task risk and move through clear levels of autonomy.

## Decide by Task Risk, Not Model Capability

Before delegating an operation, assess four properties:

- **Reversibility:** Can the action be undone quickly and completely?
- **Blast radius:** How many services, users or business processes could be affected?
- **Verifiability:** Can a person or automated check confirm the result?
- **Sensitivity:** Does it involve customer data, security settings, production configuration or money?

A read-only health check is usually a better starting point than a production configuration change. A bounded restart may be recoverable; deleting data may not be.

The same agent can be suitable for one task and unsuitable for another. Grant permissions at the task level instead of giving an agent broad authority.

This makes autonomy a decision grounded in operational risk and evidence, rather than an assumption based on model capability.

## Four Levels of Operational Autonomy

Each level should build on clear execution visibility, enforceable permissions, verifiable outcomes and accountable operation.

### Level 0: Observe

The agent reads operational data, summarizes system conditions, correlates signals and answers questions. It does not change the environment.

Examples include health summaries, alert explanations and retrieval of incident information.

### Level 1: Recommend

The agent investigates and proposes a diagnosis or action with supporting evidence. A human makes the decision.

Examples include root-cause analysis drafts, suggested investigation steps and remediation plans.

### Level 2: Act with Approval

The agent prepares an action and executes it only after an authorized person explicitly approves it.

Examples include a service restart, a bounded scaling change or a reviewed operational command.

### Level 3: Act Within Defined Bounds

The agent performs pre-approved, reversible, low-impact actions without requesting approval every time. Permissions and policy enforce the boundaries.

This is not unrestricted autonomy: action scope, monitoring, verifiability and accountability remain explicit.

## What Should an Operations Team Delegate First?

| Task | Starting Point | Reason |
|---|---|---|
| Routine inspection and health checks | Observe | Read-only and easy to verify |
| Alert triage and correlation | Recommend | Saves investigation time while keeping decisions with people |
| Root-cause analysis drafts | Recommend | Human reviewers can check evidence |
| Log and trace investigation | Observe / Recommend | Read-only analysis is reviewable |
| Scaling within limits | Act with approval | Requires validated bounds and rollback |
| Restarting stateless services | Act with approval | May be reversible, but affects production |
| Configuration and deployment changes | Human approval | Can have wider impact |
| Schema changes, data deletion and security changes | Human approval | Sensitive or difficult to reverse |

These are examples, not universal rules. A restart that is safe in one environment may cause an outage in another. Assess dependencies and failure modes before assigning the task.

## Promotion Gates: What Must Be True Before Increasing Autonomy?

Moving an agent from recommending to acting should be an explicit decision based on a defined review period.

Require:

1. Recommendation acceptance meets a team-defined target.
2. Each recommendation links to inspectable traces, metrics, logs or other evidence.
3. A reviewer can independently verify the diagnosis and expected result.
4. Rollback has been tested for every action type.
5. The agent can use only the tools and permissions assigned to it.
6. The action scope is technically restricted, not merely described in a prompt.
7. Sensitive actions require authorized approval.
8. The audit trail records actor, tool, arguments, approver, timestamp and outcome.
9. A named person or team is accountable.
10. Monitoring can detect failures and unexpected action patterns.

If a critical gate is missing, keep the agent at its current level. There is no benefit in increasing autonomy faster than the team can verify and govern it.

## Measure Outcomes, Not Autonomy Percentage

The percentage of tasks completed without human involvement can reward risky behavior. An agent that executes many actions but frequently needs overrides or rollbacks is not necessarily helping.

Track:

- Time from alert to diagnosis.
- Recommendation acceptance rate.
- Human override rate.
- Action success rate.
- Rollback rate.
- Evidence coverage.
- Blocked action attempts.
- Cost per investigation.

For a read-only assistant, investigation time and evidence quality may matter most. For an agent that changes production, action success, rollback and auditability become critical.

The goal is not simply to automate more tasks. It is to deliver useful operational outcomes that teams can inspect, verify and take responsibility for.

## Observability Is a Prerequisite for Trustworthy Agent Governance

An operations team cannot govern what it cannot inspect. To review an agent’s behavior, teams need visibility into sessions, agent steps, model calls, tool invocations, action outcomes and the operational data supporting a diagnosis.

The team should be able to answer:

- What task was the agent trying to complete?
- Which steps did it take?
- Which tools did it call?
- What did those tools return?
- Where did latency or failure occur?
- What evidence supported the recommendation?
- Who approved the action?
- What changed after execution?

OpenTelemetry can help make instrumentation portable across supported components. Coverage depends on the frameworks and instrumentation available.

Avoid collecting every prompt and payload by default: define necessary fields, mask sensitive content and restrict access.

Observability is not just a troubleshooting capability. It provides the execution evidence teams need to review agent behavior, verify outcomes and establish accountability.

## An Example 90-Day Rollout

**Days 1–30: Observe**

- Select one or two bounded workflows and assign owners.
- Instrument sessions, traces, model calls and tools where supported.
- Start with read-only inspection and alert analysis.
- Establish baseline investigation time, errors and token usage.

**Days 31–60: Recommend**

- Ask the agent to draft diagnoses and remediation plans.
- Require evidence for every conclusion.
- Compare recommendations with human investigations.
- Track acceptance, overrides and investigation time.
- Test permission boundaries and audit records.

**Days 61–90: Act with Approval**

- Select one reversible, low-impact action.
- Require explicit approval.
- Test rollback and failure handling.
- Review action records and operational impact.
- Consider bounded autonomy only if all promotion gates are met.

Finishing the 90 days at the recommendation level can be a valid result. The goal is safe operational value, not reaching the highest autonomy level on a schedule.

## Where Bonree ONE Fits

Bonree ONE provides an AI agent workbench through Sage AI, with approval policies and audit logging for agent operations. These controls can support teams in reviewing and governing agent actions as they move from recommendations to approved execution.

The autonomy levels in this article are a general adoption framework, not a product-specific maturity model.

## The Takeaway

Agentic AI operations should expand through evidence, not enthusiasm.

Start with read-only work. Require traceable evidence for recommendations. Add human-approved actions only after permissions and rollback have been tested. Consider bounded autonomy when the task is reversible, the impact is limited and results are consistently verifiable.

**Trustworthy autonomy is built through visibility, control and verifiable evidence — not assumed from model capability.**

---
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>observability</category>
      <category>automation</category>
    </item>
    <item>
      <title>Choosing an AI Observability Platform for Financial Services: 8 Things to Evaluate Before Production</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Tue, 03 Nov 2026 08:35:21 +0000</pubDate>
      <link>https://dev.to/bonree-js/choosing-an-ai-observability-platform-for-financial-services-8-things-to-evaluate-before-production-2kk5</link>
      <guid>https://dev.to/bonree-js/choosing-an-ai-observability-platform-for-financial-services-8-things-to-evaluate-before-production-2kk5</guid>
      <description>&lt;p&gt;Financial institutions are bringing LLM applications and AI agents into customer service, internal knowledge workflows and operational processes. Once an AI application reaches production, monitoring model availability is not enough.&lt;/p&gt;

&lt;p&gt;A user may report that an assistant gave the wrong answer. An engineer may see response time increasing. A platform team may notice token consumption growing faster than request volume. These symptoms may come from different parts of the same execution path.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A useful AI observability platform connects the user interaction to the agent workflow, model calls, retrieval, tool execution and the application services underneath.&lt;/strong&gt; It should also provide enough context to investigate what happened without exposing more sensitive data than necessary.&lt;/p&gt;

&lt;p&gt;Here are eight areas to evaluate.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, understand what you need to observe
&lt;/h2&gt;

&lt;p&gt;A production AI request may cross several layers: session, agent workflow, LLM calls, retrieval, tools, application services and infrastructure. If these signals live in separate systems, an investigation becomes a manual correlation exercise.&lt;/p&gt;

&lt;p&gt;The goal is to make the execution path understandable.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Can you trace an AI request end to end?
&lt;/h2&gt;

&lt;p&gt;A request may involve multiple model calls, retrieval operations, tool invocations and downstream requests. A single model-call log does not explain the full workflow.&lt;/p&gt;

&lt;p&gt;Check for Trace and Span relationships, timing, status, useful attributes, visibility into model/retrieval/tool operations and support for the frameworks your teams use.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Demo test:&lt;/strong&gt; Give the vendor one slow AI request. Can they show the full path from the user request to the slow model call, retrieval step or downstream API?&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Can you investigate a conversation, not just a request?
&lt;/h2&gt;

&lt;p&gt;Earlier turns may establish context. A tool call may change state. Later turns may depend on information gathered earlier.&lt;/p&gt;

&lt;p&gt;Session-level analysis should group related requests, link a session to its traces and help find recurring failures or unusually long interactions. A request trace explains one execution; session analysis adds the wider conversation context.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Can you understand model usage and token growth?
&lt;/h2&gt;

&lt;p&gt;Token consumption can change because traffic increases, prompts become longer, retrieved context grows, a different model is selected or failed workflows trigger retries.&lt;/p&gt;

&lt;p&gt;Evaluate input, output and total token visibility, breakdowns by model/service/time, and correlation with request volume, latency and errors.&lt;/p&gt;

&lt;p&gt;Do not treat a token chart as a complete cost model. Actual cost may depend on provider pricing, model type, cached tokens and billing rules. Clarify which values are measured, estimated or imported.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Can you see what the agent did with its tools?
&lt;/h2&gt;

&lt;p&gt;Suppose an assistant answers a delivery question. It retrieves an order record, calls a logistics API and summarizes the result. If the answer is wrong, the cause may be stale retrieval data, a failed API call, an unexpected tool response or a model interpretation issue.&lt;/p&gt;

&lt;p&gt;Check whether you can see tool name, timing, status, agent handoffs, relevant metadata and the downstream service involved.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Where does telemetry go, and what sensitive data is collected?
&lt;/h2&gt;

&lt;p&gt;Prompts, responses, retrieved content and tool arguments may contain customer or business information.&lt;/p&gt;

&lt;p&gt;Ask where telemetry is stored and processed, which deployment options exist, whether fields can be masked or excluded, who can access detailed traces, and how retention, deletion and export work.&lt;/p&gt;

&lt;p&gt;The objective is not to collect everything. It is to collect enough evidence to operate the application responsibly.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Can you verify the platform’s explanations?
&lt;/h2&gt;

&lt;p&gt;If a platform surfaces a diagnostic suggestion, the explanation should lead back to evidence: the relevant trace or Span, associated metric or log, timestamp, request/session context and observed error.&lt;/p&gt;

&lt;p&gt;A conclusion that cannot be checked against telemetry is difficult to use in a serious incident review.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Does it fit your existing engineering environment?
&lt;/h2&gt;

&lt;p&gt;Evaluate OpenTelemetry and existing instrumentation, APM and infrastructure monitoring, incident management, ITSM, identity integration, APIs and data export.&lt;/p&gt;

&lt;p&gt;Avoid creating a new silo for AI telemetry. Engineers should be able to move from an AI trace to the service, database or infrastructure evidence behind it.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Is the platform production-ready?
&lt;/h2&gt;

&lt;p&gt;Ask about availability, scaling, data handling during interruptions, upgrades, access control and support. Clarify how data volume affects retention and cost. AI workloads can generate many spans and high-cardinality attributes; a small proof of concept does not prove production scalability.&lt;/p&gt;

&lt;h2&gt;
  
  
  A vendor evaluation exercise
&lt;/h2&gt;

&lt;p&gt;Use the same scenario for every shortlisted vendor:&lt;/p&gt;

&lt;p&gt;An AI assistant takes 12 seconds to answer a customer question and returns an incorrect result. Some requests retry, token consumption has increased, but the application service appears healthy.&lt;/p&gt;

&lt;p&gt;Ask each vendor to demonstrate how to find the session, inspect the full trace, identify latency contributors, check tool failures and retries, review token usage, inspect data masking, correlate application/infrastructure signals and export the investigation.&lt;/p&gt;

&lt;p&gt;This tests the actual investigation workflow, not just dashboard design.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to start without overengineering
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Choose an application with a clear owner and measurable user impact.&lt;/li&gt;
&lt;li&gt;Establish baseline request volume, latency, errors and token usage.&lt;/li&gt;
&lt;li&gt;Instrument sessions, traces, model calls, retrieval and tools where supported.&lt;/li&gt;
&lt;li&gt;Test slow responses, wrong answers, tool failures, retries and token spikes.&lt;/li&gt;
&lt;li&gt;Validate access, masking, retention and deployment requirements.&lt;/li&gt;
&lt;li&gt;Expand after engineering and operations teams find the evidence useful.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Where Bonree ONE fits
&lt;/h2&gt;

&lt;p&gt;Bonree ONE is one example of a platform combining AI observability with application and infrastructure monitoring. Its AI observability capabilities provide visibility into LLM and AI agent call chains, sessions and token consumption, helping teams examine AI execution alongside the services and infrastructure involved.&lt;/p&gt;

&lt;p&gt;The right choice still depends on your own frameworks, architecture, data policies and operating requirements. Use the same real-world scenario to validate every platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;For financial services, AI observability is not just about seeing whether a model call succeeded. It is about understanding how an AI interaction was executed, what affected its performance, which tools and services were involved, and how much resource it consumed.&lt;/p&gt;

&lt;p&gt;Start with the workflow you care about. Ask vendors to trace it end to end. Choose the platform that can demonstrate the evidence your teams actually need.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>observability</category>
      <category>opentelemetry</category>
      <category>devops</category>
    </item>
    <item>
      <title>Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Fri, 30 Oct 2026 04:10:22 +0000</pubDate>
      <link>https://dev.to/bonree-js/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai-observability-1cjp</link>
      <guid>https://dev.to/bonree-js/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai-observability-1cjp</guid>
      <description>&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Tracing&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;AI&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Agents&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;with&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;OpenTelemetry:&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;A&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Design&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;for&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;LLM&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Agentic&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;AI&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Observability"&lt;/span&gt;
&lt;span class="na"&gt;published&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;run-centric&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;trace&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;model&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;for&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;LLM&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;agentic&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;AI&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;observability&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;with&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;OpenTelemetry&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;GenAI&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;conventions,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;with&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;measurements&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;from&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;example."&lt;/span&gt;
&lt;span class="na"&gt;tags&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;opentelemetry, observability, devops, bonree&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;LLM applications and agentic AI operations share a property that classic APM was not designed for: the execution path is decided at runtime by a model. A request that returns HTTP 200 may have looped, retried a failing tool, or consumed several times the usual number of tokens, and none of that is visible from the response.&lt;/p&gt;

&lt;p&gt;This article argues for a run-centric trace model, implements it with the OpenTelemetry GenAI semantic conventions (&lt;code&gt;gen_ai.*&lt;/code&gt;), and reports what the trace contains when an example agent is executed. It then covers the production trade-offs that decide whether such tracing survives real traffic: sampling, cardinality, content capture and context propagation across agents.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Conclusion.&lt;/strong&gt; One trace per run, three span types (&lt;code&gt;invoke_agent&lt;/code&gt;, &lt;code&gt;chat&lt;/code&gt;, &lt;code&gt;execute_tool&lt;/code&gt;), token usage on every model-call span and error status on every failed tool call are sufficient to answer three of the four questions that recur in agent incidents: which path was taken, what it cost, and what was changed. The fourth, whether the answer was correct, needs an evaluation signal that tracing cannot supply.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why request-centric tracing falls short
&lt;/h2&gt;

&lt;p&gt;Four properties distinguish an agent run from a request:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The path is chosen at runtime.&lt;/strong&gt; The agent plans, calls a tool, observes, plans again and may hand off to another agent. The shape of the trace varies from run to run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost is driven by tokens,&lt;/strong&gt; which depend on loop count and context size, not on request count.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Actions have side effects.&lt;/strong&gt; A tool call can update a record or issue a refund, which a model-call span cannot reveal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failure can look like success.&lt;/strong&gt; A fast, well-formed, wrong answer produces no error and no latency anomaly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The consequence is that the entity to observe is the run, and its internal structure matters more than its endpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  A run-centric trace model
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Span name&lt;/th&gt;
&lt;th&gt;&lt;code&gt;gen_ai.operation.name&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Represents&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;invoke_agent {agent}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;invoke_agent&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The whole run&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;chat {model}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;chat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;One model request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;execute_tool {tool}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;execute_tool&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;One tool invocation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three design choices are worth stating explicitly.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Model calls and tool calls are &lt;strong&gt;child spans&lt;/strong&gt;, not events on the run, because they have their own duration, status and attributes, which events lack.&lt;/li&gt;
&lt;li&gt;Retries are &lt;strong&gt;events on the parent&lt;/strong&gt;, because a retry is a decision of the parent and not an operation with its own cost.&lt;/li&gt;
&lt;li&gt;Anything outside the specification goes under an application-owned namespace (&lt;code&gt;app.*&lt;/code&gt;), so that a later addition to the specification cannot collide with it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Reference implementation
&lt;/h2&gt;

&lt;p&gt;The listing is complete and runs without an API key. The model client and the tools are stubs, and the refund tool is configured to fail once and then succeed, so every run is identical.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;opentelemetry&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;opentelemetry.sdk.resources&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Resource&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;opentelemetry.sdk.trace&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;TracerProvider&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;opentelemetry.sdk.trace.export&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BatchSpanProcessor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ConsoleSpanExporter&lt;/span&gt;

&lt;span class="n"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;TracerProvider&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;Resource&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;service.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;support-agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}))&lt;/span&gt;
&lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_span_processor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;BatchSpanProcessor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ConsoleSpanExporter&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_tracer_provider&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;tracer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_tracer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;support-agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;PROVIDER&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;example-provider&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;example-model&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call_llm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Stand-in for a real model client.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;...&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;finish_reason&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stop&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;traced_chat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;tracer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start_as_current_span&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chat &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.operation.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chat&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.provider.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;PROVIDER&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.request.model&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;call_llm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.usage.input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.usage.output_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.response.finish_reasons&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;finish_reason&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;traced_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# An exception raised inside the span is recorded and the span status is set
&lt;/span&gt;    &lt;span class="c1"&gt;# to ERROR automatically by the SDK; we only add the semantic-convention
&lt;/span&gt;    &lt;span class="c1"&gt;# error.type attribute so failures can be grouped by class.
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;tracer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start_as_current_span&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;execute_tool &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.operation.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;execute_tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.tool.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error.type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;lookup_order&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;delivered&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="n"&gt;_transient_failures&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;issue_refund&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;  &lt;span class="c1"&gt;# simulate one timeout, then success
&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;issue_refund&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;_transient_failures&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;issue_refund&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;_transient_failures&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;issue_refund&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;TimeoutError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;upstream refund API timed out&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refunded&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_agent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;tracer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start_as_current_span&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;invoke_agent support-agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.operation.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;invoke_agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.provider.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;PROVIDER&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.agent.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;support-agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="n"&gt;model_calls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="nf"&gt;traced_chat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;model_calls&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="nf"&gt;traced_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lookup_order&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lookup_order&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A-1042&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;traced_chat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order found, decide next step&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;model_calls&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="nf"&gt;traced_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;issue_refund&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;issue_refund&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A-1042&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;break&lt;/span&gt;
            &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TimeoutError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool_retry&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;issue_refund&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;attempt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;

        &lt;span class="nf"&gt;traced_chat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;write the final answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;model_calls&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;app.agent.model_calls&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model_calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# custom attribute
&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;run_agent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Where is my order A-1042, and can I get a refund?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four notes on the listing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Errors.&lt;/strong&gt; With default SDK behavior, an exception that propagates out of a span is recorded as an event and sets the span status to &lt;code&gt;ERROR&lt;/code&gt;. The only addition is &lt;code&gt;error.type&lt;/code&gt;, which makes failures groupable by class.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tokens stay on the span that produced them.&lt;/strong&gt; Aggregates per run, per agent and per model can be derived later. The reverse is not possible.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Finish reasons are recorded&lt;/strong&gt; because a truncated or filtered output otherwise reads as success.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompts and completions are not stored.&lt;/strong&gt; The reasons are covered under production trade-offs below.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What the measurements show
&lt;/h2&gt;

&lt;p&gt;Running the listing produces seven spans: one &lt;code&gt;invoke_agent&lt;/code&gt;, three &lt;code&gt;chat&lt;/code&gt; and three &lt;code&gt;execute_tool&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Span&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;chat example-model&lt;/code&gt; (1st)&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;22 input / 48 output tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;execute_tool lookup_order&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;chat example-model&lt;/code&gt; (2nd)&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;10 input / 48 output tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;execute_tool issue_refund&lt;/code&gt; (1st)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;ERROR&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;error.type = TimeoutError&lt;/code&gt;, exception event recorded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;execute_tool issue_refund&lt;/code&gt; (2nd)&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;The retry succeeded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;chat example-model&lt;/code&gt; (3rd)&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;8 input / 48 output tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;invoke_agent support-agent&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;tool_retry&lt;/code&gt; event; &lt;code&gt;app.agent.model_calls = 3&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2FREPLACE-WITH-UPLOADED-URL%2Fdevto-trace-waterfall.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2FREPLACE-WITH-UPLOADED-URL%2Fdevto-trace-waterfall.png" alt="One agent run, measured: seven spans from the example script" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Three observations follow directly from this data.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The run consumed 40 input and 144 output tokens, so cost per run is the sum over &lt;code&gt;chat&lt;/code&gt; spans.&lt;/li&gt;
&lt;li&gt;The run succeeded, yet the trace still records one failed refund attempt. A metric that only counts outcomes would report this run as fully healthy.&lt;/li&gt;
&lt;li&gt;The retry is an event on the parent span, so retries can be counted per agent without searching logs.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In this example the model stub dominates run time by construction. In a real system, the split between model time and tool time is exactly what the trace is meant to reveal.&lt;/p&gt;

&lt;h2&gt;
  
  
  Production trade-offs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Sampling.&lt;/strong&gt; Head-based sampling decides before a run starts and therefore discards the rare runs that matter: failures, loops and unusually expensive runs. Tail-based sampling (for example, the OpenTelemetry Collector's tail sampling processor) decides after the run completes. It can keep every run with an error status, a high model-call count or high token usage, plus a small fraction of ordinary runs as a baseline.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cardinality.&lt;/strong&gt; Identifiers such as conversation IDs or order numbers belong on spans, not on metric dimensions. Derive metrics from spans with bounded labels: agent name, model, tool name and &lt;code&gt;error.type&lt;/code&gt;. The GenAI conventions also define client metrics for token usage and operation duration, which fit this pattern.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Content capture.&lt;/strong&gt; Prompts and completions are large and often sensitive. Keep capture off by default, enable it per environment behind redaction, and prefer storing a reference to the payload over the payload itself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Context propagation across agents.&lt;/strong&gt; When one agent hands work to another over HTTP or a message queue, propagate the W3C trace context in the request headers or message metadata. Otherwise the run fragments into disconnected traces, and the seam, where multi-agent failures tend to occur, becomes the part that cannot be seen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Specification drift.&lt;/strong&gt; Parts of the GenAI conventions are still experimental. Keep attribute names in one module so that a rename is a one-line change, and pin the instrumentation version.&lt;/p&gt;

&lt;h2&gt;
  
  
  From tracing to agentic AI operations
&lt;/h2&gt;

&lt;p&gt;Agents that operate production systems, for example to perform root cause analysis, are both consumers and subjects of observability data.&lt;/p&gt;

&lt;p&gt;As consumers, their conclusions should reference the telemetry they rely on (topology, metrics, errors), so that an engineer can verify a diagnosis instead of repeating it. As subjects, their own runs need the trace structure above, plus a record of which actions were proposed, approved and executed. Determinism cannot be demanded of the model, but it can be demanded of the system around it: pinned model and prompt versions, typed tools with narrow permissions, and replayable runs.&lt;/p&gt;

&lt;p&gt;Bonree ONE's AI Observability applies this model to applications built on LangChain, LangGraph, Dify and OpenClaw, and places the agent data next to the services beneath it. Bonree ONE Sage AI, Bonree's agent operations platform, builds its fault diagnosis on topology, metric and error data of the kind described here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Tracing records behavior, not correctness. A run can be fast, error-free and wrong, so an evaluation signal (offline test set, online judge or human review) has to be attached to the run.&lt;/li&gt;
&lt;li&gt;The example uses stubs, so it shows structure, not realistic latency distributions or token counts.&lt;/li&gt;
&lt;li&gt;The GenAI conventions are still evolving, and attribute names may change.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusions
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Model an agent run as a tree: &lt;code&gt;invoke_agent&lt;/code&gt; with &lt;code&gt;chat&lt;/code&gt; and &lt;code&gt;execute_tool&lt;/code&gt; children.&lt;/li&gt;
&lt;li&gt;Record tokens, finish reasons, tool names and error status on every span.&lt;/li&gt;
&lt;li&gt;Use tail-based sampling so that failures and expensive runs are always kept.&lt;/li&gt;
&lt;li&gt;Keep identifiers on spans and bounded labels on metrics.&lt;/li&gt;
&lt;li&gt;Propagate trace context across agent hand-offs.&lt;/li&gt;
&lt;li&gt;Add an evaluation signal, because latency and errors cannot establish correctness.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A broader checklist for reviewing agent instrumentation is available at &lt;a href="https://github.com/Bonree-official/Technology-Blog/blob/main/Agentic-AI-Operations-Evidence-Carrying-Handoffs.md" rel="noopener noreferrer"&gt;agent-trace-checklist&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: I am part of the Bonree team. The code above works with any OTLP-compatible backend.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>discuss</category>
      <category>sql</category>
    </item>
    <item>
      <title>Choosing an AI Observability Platform in 2026: Four Technical Routes, and How to Avoid Lock-in</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Thu, 29 Oct 2026 03:01:28 +0000</pubDate>
      <link>https://dev.to/bonree-js/choosing-an-ai-observability-platform-in-2026-four-technical-routes-and-how-to-avoid-lock-in-4ago</link>
      <guid>https://dev.to/bonree-js/choosing-an-ai-observability-platform-in-2026-four-technical-routes-and-how-to-avoid-lock-in-4ago</guid>
      <description>&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Choosing&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;an&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;AI&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Observability&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Platform&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;in&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2026:&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Four&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Technical&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Routes,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;How&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;to&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Avoid&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Lock-in"&lt;/span&gt;
&lt;span class="na"&gt;published&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Full-stack&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;platforms,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;LLM-native&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;tools,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;OpenTelemetry-native&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;stacks&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;AI&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;agent&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;operations&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;layers&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;compared&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;by&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;how&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;gets&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;in,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;what&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;each&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;is&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;good&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;at,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;where&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;lock-in&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;hides."&lt;/span&gt;
&lt;span class="na"&gt;tags&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;observability, ai, opentelemetry, devops&lt;/span&gt;
&lt;span class="na"&gt;cover_image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UPLOAD_otel-genai-instrument-once.png_AND_PASTE_URL_HERE&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you ship LLM apps or AI agents, you've probably noticed that "AI observability" means different things in different tools. This post sorts the market into four technical routes, compares them on practical dimensions, and ends with a small piece of code that keeps your options open. Product facts come from vendors' public documentation as of October 2026; check current docs before you decide.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two jobs hiding under one name
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Observing your AI apps:&lt;/strong&gt; traces for model calls, tool calls and agent steps, plus token cost, latency and quality.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Using AI to run operations:&lt;/strong&gt; agents that investigate incidents, run inspections and propose fixes.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The four routes
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Full-stack platform with an AI module
&lt;/h3&gt;

&lt;p&gt;AI spans live next to service and infrastructure traces. Datadog's Agent Observability documents SDK auto-instrumentation (18 Python frameworks and libraries, including LangChain, LangGraph, CrewAI and OpenAI Agents) and also supports frameworks that natively emit OpenTelemetry GenAI spans (v1.37+). Dynatrace takes telemetry through OneAgent, OTLP or OpenInference and runs online LLM-as-a-judge checks on production traces. New Relic announced GA of OpenTelemetry-based AI monitoring on August 31, 2026, with no New Relic agent required and prompt/completion capture off by default.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pick it when&lt;/strong&gt; you already run the platform and want AI latency and errors correlated with everything else. &lt;strong&gt;Check&lt;/strong&gt; evaluation depth and the exact framework list.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. LLM-native engineering platform
&lt;/h3&gt;

&lt;p&gt;Built around the LLM development loop. Langfuse, for example, describes itself as an open-source platform for tracing, prompt management, evals and experiments, can be self-hosted (Docker Compose, Kubernetes, Terraform), and accepts OTLP.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pick it when&lt;/strong&gt; you iterate on prompts and evaluations daily or must keep data in your environment. &lt;strong&gt;Check&lt;/strong&gt; how you will correlate with infrastructure monitoring.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. OpenTelemetry-native open stack
&lt;/h3&gt;

&lt;p&gt;Instrument with open standards and send to any OTLP backend. Grafana Cloud's AI Observability uses the OpenLIT SDK: one &lt;code&gt;openlit.init()&lt;/code&gt; call, traces in Tempo, token and cost metrics in Prometheus, prebuilt dashboards.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pick it when&lt;/strong&gt; you have platform engineers and want portable data. &lt;strong&gt;Check&lt;/strong&gt; that evaluation and dashboards are largely yours to build.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. AI agent operations layer
&lt;/h3&gt;

&lt;p&gt;Dynatrace Intelligence (an agentic operations system), Datadog Bits AI SRE (an autonomous agent that investigates alerts) and Splunk AI SRE (investigates and drafts guided remediation plans, with engineers kept in the loop) sit in this route. So does Bonree ONE · Sage AI.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pick it when&lt;/strong&gt; you have repetitive operations work. &lt;strong&gt;Check&lt;/strong&gt; trust: evidence, approval boundaries, audit trail, and replay on your own incidents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Comparison
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimension&lt;/th&gt;
&lt;th&gt;1. Full-stack&lt;/th&gt;
&lt;th&gt;2. LLM-native&lt;/th&gt;
&lt;th&gt;3. OTel-native&lt;/th&gt;
&lt;th&gt;4. Agent ops&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Data in&lt;/td&gt;
&lt;td&gt;SDK/agent + OTLP&lt;/td&gt;
&lt;td&gt;SDK + OTLP&lt;/td&gt;
&lt;td&gt;OTel GenAI + any OTLP backend&lt;/td&gt;
&lt;td&gt;Existing observability data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Correlation with infra&lt;/td&gt;
&lt;td&gt;Strong&lt;/td&gt;
&lt;td&gt;You build it&lt;/td&gt;
&lt;td&gt;Backend dependent&lt;/td&gt;
&lt;td&gt;Platform dependent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Evaluations&lt;/td&gt;
&lt;td&gt;Varies&lt;/td&gt;
&lt;td&gt;Core strength&lt;/td&gt;
&lt;td&gt;Mostly DIY&lt;/td&gt;
&lt;td&gt;Not the focus&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Portability&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;Medium-high&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;(Ratings are my judgment, not vendor claims.)&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep your options open: emit &lt;code&gt;gen_ai.*&lt;/code&gt; spans yourself
&lt;/h2&gt;

&lt;p&gt;Whichever route you choose, you can instrument once with OpenTelemetry and swap the backend by changing configuration.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;opentelemetry&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;

&lt;span class="n"&gt;tracer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_tracer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;my-llm-app&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# Span name convention: "&amp;lt;operation&amp;gt; &amp;lt;model&amp;gt;"
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;tracer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start_as_current_span&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chat &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SpanKind&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CLIENT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.operation.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chat&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.provider.name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;my-provider&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.request.model&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;my_client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# your SDK call
&lt;/span&gt;        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.usage.input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_tokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gen_ai.usage.output_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Point the exporter at your backend with the standard OTLP environment variables (for example &lt;code&gt;OTEL_EXPORTER_OTLP_ENDPOINT&lt;/code&gt;), and the same spans can go to a full-stack platform, an LLM-native tool or an open-source backend.&lt;/p&gt;

&lt;p&gt;Three caveats:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The &lt;code&gt;gen_ai.*&lt;/code&gt; conventions are still marked Development. Names have changed (&lt;code&gt;gen_ai.system&lt;/code&gt; → &lt;code&gt;gen_ai.provider.name&lt;/code&gt;; &lt;code&gt;prompt_tokens&lt;/code&gt;/&lt;code&gt;completion_tokens&lt;/code&gt; → &lt;code&gt;input_tokens&lt;/code&gt;/&lt;code&gt;output_tokens&lt;/code&gt;). Pin versions and check the current spec.&lt;/li&gt;
&lt;li&gt;Prompt and completion content is not captured by default. Decide what you store.&lt;/li&gt;
&lt;li&gt;The data moves easily; evaluation datasets, judges and agent workflows tend to stay inside one product.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Trust checklist for AI that operates
&lt;/h2&gt;

&lt;p&gt;If you pick route 4 (or any AI that acts for you), ask: Can every conclusion be traced to evidence? Are facts fetched from reliable sources? What can the agent do alone versus with approval? Can you audit its steps? Have you replayed your own past incidents?&lt;/p&gt;

&lt;p&gt;&lt;em&gt;(Disclosure: I'm on the Bonree team. Bonree ONE is a unified observability platform with built-in AI observability for LLM apps, and Bonree ONE · Sage AI is its AI Agent operations platform, which supports human-in-the-loop approval and audit logging.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Learn more: &lt;a href="https://en.bonree.com" rel="noopener noreferrer"&gt;https://en.bonree.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>agents</category>
    </item>
    <item>
      <title>AI Agent Observability: How to Trace Multi-Turn Conversations, Tool Calls and Agent Workflows" published: false</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Fri, 23 Oct 2026 04:09:02 +0000</pubDate>
      <link>https://dev.to/bonree-js/ai-agent-observability-how-to-trace-multi-turn-conversations-tool-calls-and-agent-workflows-1730</link>
      <guid>https://dev.to/bonree-js/ai-agent-observability-how-to-trace-multi-turn-conversations-tool-calls-and-agent-workflows-1730</guid>
      <description>&lt;p&gt;AI agent observability is the practice of recording what an AI agent received, did and returned across a whole conversation, including model calls, tool calls, retrieved context, token usage, latency and errors, so that a team can debug the agent, control its cost and verify its behavior. This guide explains what to record, why the session is the right unit of analysis, and how to keep traces trustworthy.&lt;br&gt;
Why isn’t traditional monitoring enough for AI agents?&lt;br&gt;
A traditional application follows a fixed path: a request reaches a service, the service reads a database, and a response comes back. An AI agent does not. One user request can trigger several model calls, external tools, knowledge retrieval and even other agents, and the path changes from one run to the next.&lt;/p&gt;

&lt;p&gt;User request&lt;br&gt;
  → Agent planning&lt;br&gt;
  → LLM call&lt;br&gt;
  → Tool execution&lt;br&gt;
  → Knowledge retrieval&lt;br&gt;
  → Another agent&lt;br&gt;
  → Final response&lt;/p&gt;

&lt;p&gt;Traditional monitoring answers “Is my application working?” LLM observability for agents has to answer a harder question: “Why did the agent give this answer?” The final answer alone does not explain what happened. When an agent fails, a team needs to see what information it received, which tools it used, what those tools returned and which decisions the model made.&lt;br&gt;
Why trace at the session level?&lt;br&gt;
A single conversation can contain several turns, and each turn can contain model calls, tool calls, retrieval steps and agent decisions. If you only look at individual requests, you see isolated model calls. You cannot see that the third turn was slow because the second turn’s tool returned a very large payload, or that cost climbed because the agent kept retrying. Both the performance problem and the cost problem live in the sequence.&lt;br&gt;
Session-level tracing connects these steps into one workflow. The attributes that make it work are a session ID, a turn index, parent/child links between steps, and hand-off markers when one agent calls another.&lt;br&gt;
What should an AI agent trace record?&lt;br&gt;
Model calls. Record the model name and version, generation parameters such as temperature, input and output tokens, latency, finish reason, errors and retries. Without these, a team cannot explain why behavior changed between two runs.&lt;br&gt;
Tool calls. Record the tool name and version, the arguments (redacted where needed), the result status and size, and the duration. Tools such as database queries, search systems and APIs are a critical part of most agents.&lt;br&gt;
Cost attribution. Record tags for the model, application, team or business line, and environment. With them you can answer which model uses the most tokens, which workflow is expensive and which agent creates unexpected cost.&lt;br&gt;
Versions. Record the prompt version, model version, framework version and tool version. A small update to any of them can change the final answer.&lt;br&gt;
Content policy. Traces can contain sensitive data, so decide for every field whether to store it, sample it, redact it or never store it. Operational metadata such as timings and token counts is usually safe to store. Secrets and personal data should not be stored.&lt;br&gt;
How do you keep traces trustworthy?&lt;br&gt;
Trust in an AI system is built on evidence. A read-only tool call, such as a database query, should return the same data for the same input, as long as the underlying data has not changed. Model output varies from run to run. If a trace records tool results and model output separately, a team can tell whether a wrong answer came from bad data, a failing tool or the model’s reasoning. That separation is what makes an agent’s behavior verifiable.&lt;br&gt;
Do frameworks change what you record?&lt;br&gt;
The checklist is the same whichever framework you use, but the effort of collecting it is not. Teams build agents in different ways: in code with LangChain, as stateful graphs with LangGraph, on low-code platforms such as Dify, or on frameworks such as OpenClaw. If every framework needs its own tracing setup, session traces become patchy. When you choose tooling, check that the frameworks your teams actually use are covered, and use shared attribute names, such as the OpenTelemetry semantic conventions for generative AI, so traces stay portable.&lt;br&gt;
From AI agent observability to AI agent operations&lt;br&gt;
Observability tells a team what happened. The next step is helping teams operate systems with the help of AI agents that use observability data to diagnose incidents, inspect system health, analyze database performance and assess capacity.&lt;br&gt;
Disclosure: I work at Bonree (博睿数据). Bonree ONE is our AI-driven observability platform. Its AI observability capability covers session-level tracing and token cost analysis for AI applications, and collects data from applications built on LangChain, LangGraph, Dify and OpenClaw. Bonree ONE·Sage AI is the intelligent operations workspace in Bonree ONE, with ready-to-use agents for scenarios such as incident diagnosis and database performance analysis, and support for creating custom agents. The checklist in this article does not depend on any product.&lt;br&gt;
&lt;strong&gt;FAQ&lt;/strong&gt;&lt;br&gt;
&lt;strong&gt;What is the difference between LLM observability and AI agent observability?&lt;/strong&gt;&lt;br&gt;
 LLM observability focuses on individual model calls. AI agent observability also follows tool calls, retrieval, hand-offs and the whole multi-turn session.Should I store every prompt and output?&lt;br&gt;
 No. Decide per field. Sampling or redaction is a common default, and secrets and personal data should not be stored.Why record tool results separately from model output?&lt;br&gt;
 Read-only tool results are repeatable and model output is not, so separating them shows whether a failure came from data, tools or reasoning.Is there a ready-made checklist?&lt;br&gt;
 Yes. The GitHub repository ai-agent-observability-checklist contains the full checklist, a content capture policy and an example trace.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>llm</category>
      <category>sre</category>
    </item>
    <item>
      <title>How Should We Evaluate AI Agent Observability Platforms in 2026? A 10-Point Scorecard</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Thu, 15 Oct 2026 11:47:05 +0000</pubDate>
      <link>https://dev.to/bonree-js/how-should-we-evaluate-ai-agent-observability-platforms-in-2026-a-10-point-scorecard-2ojm</link>
      <guid>https://dev.to/bonree-js/how-should-we-evaluate-ai-agent-observability-platforms-in-2026-a-10-point-scorecard-2ojm</guid>
      <description>&lt;p&gt;An AI agent observability platform records and analyzes what AI agents do in production, including sessions, model calls, tool calls, token cost and versions, so that teams can debug agents, control cost and verify results; in 2026, evaluating one means testing ten criteria on your own workloads instead of comparing feature lists. This guide gives you those criteria and a two-week proof-of-concept plan. It covers LLM observability tools, AI observability platforms and the newer AIOps platforms that add AI agent operations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why are feature lists a poor way to compare platforms?&lt;/strong&gt;&lt;br&gt;
Most AI observability and LLM observability tools claim tracing, token tracking and dashboards. The differences show up only when you run your own agents through them: a framework that is “supported” only through manual work, a session that cannot be opened as one unit, or a cost report that cannot be split by team. A useful evaluation turns each claim into a test.&lt;br&gt;
&lt;strong&gt;What are the ten criteria?&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Session-level tracing. An agent conversation has several turns, and each turn has model calls, tool calls and sometimes hand-offs. Test: replay a five-turn conversation and follow it from start to finish.&lt;/li&gt;
&lt;li&gt;Call detail. The platform should record model, parameters, tokens, latency, errors and retries for model calls, and tool name, arguments, result status and duration for tool calls. Test: pick one slow turn and find the step that caused it.&lt;/li&gt;
&lt;li&gt;Framework coverage. Ask which frameworks are supported, at what depth (automatic or manual), and in which versions, and whether OpenTelemetry ingestion is available as a fallback. Test: list the frameworks you run today and expect to run in 12 months, and try each one.&lt;/li&gt;
&lt;li&gt;Evidence and determinism. Trustworthy AI depends on evidence. A read-only tool call should return the same data for the same input, and model output varies, so a platform should record them separately. Test: take a known wrong answer and check whether the trace shows where it went wrong.&lt;/li&gt;
&lt;li&gt;Cost attribution. Token cost should roll up by model, application, team, business line and time. Test: find your most expensive workflow in under ten minutes.&lt;/li&gt;
&lt;li&gt;Quality evaluation. Check whether the platform offers evaluators, annotation or datasets, or integrates with a tool that does. Test: score a sample of real outputs.&lt;/li&gt;
&lt;li&gt;Data governance. You should be able to choose per field whether to store, sample, redact or never store content, and to understand retention and data-location options. Test: send a prompt containing fake personal data and see what is kept.&lt;/li&gt;
&lt;li&gt;Context. AI traces should link to the services, infrastructure and users underneath. Test: follow a slow answer down to a database or service problem.&lt;/li&gt;
&lt;li&gt;From insight to action. Some platforms now add AI agent operations, where agents diagnose incidents, inspect systems and assess risk on observability data. Ask what the agents can do alone, what needs human approval, and whether every step is logged. Score this N/A if you only need observability.&lt;/li&gt;
&lt;li&gt;Pricing and scale. Find out what is billed (spans, tokens, hosts or seats) and estimate the monthly bill for your real traffic.
How should you read framework coverage?
A longer list is not automatically better. What matters is whether the frameworks your teams use are covered, and how deeply. Teams build agents in different ways: in code with LangChain, as stateful graphs with LangGraph, on low-code platforms such as Dify, or on other frameworks such as OpenClaw. A platform that covers several of these build styles lets different teams in one company share the same tracing, instead of running a separate setup for each. When a framework is missing from a list, ask whether OpenTelemetry ingestion covers it and what you must configure.
How do you test whether an AI platform is trustworthy?
Whether the platform records AI behavior or runs AI agents of its own, ask the same questions. Can each conclusion be traced to the data it came from? Are the facts the AI works from retrieved by defined queries, so the same question returns the same data? Are actions bounded by permissions or approval? Is every step logged? Does it hold up on your past incidents? Vendors can describe all of this, but only a test on your own incidents shows it.
What does a two-week proof of concept look like?
Week 1. Instrument two representative agents, ideally built with different frameworks. Replay three real incidents where an agent gave a wrong or slow answer, and note how long it takes to find the cause.
Week 2. Roll up token cost by application and team and compare it with your provider invoice. Test the content policy with fake sensitive data. Run realistic traffic and check overhead and the estimated bill.
Where does Bonree ONE fit?
Disclosure: I work at Bonree (博睿数据). Bonree ONE is our AI-driven observability platform. Its AI observability capability covers session-level tracing and token cost analysis by model, business line and time, and collects data from applications built on LangChain, LangGraph, Dify and OpenClaw. For AI agent operations, Bonree ONE·Sage AI provides agents for scenarios such as incident diagnosis, database performance analysis, system health inspection, capacity assessment and change risk, and supports custom agents. Use the scorecard on our platform the same way you would on any other.
FAQ
What is an AI agent observability platform?
A platform that records and analyzes sessions, model calls, tool calls, cost and versions of AI agents in production.How is it different from traditional APM?
APM measures requests through services. AI agent observability adds prompts, outputs, tokens and multi-turn sessions, and follows tool calls and agent hand-offs.Do I need a dedicated LLM tool, or can my observability platform do it?
It depends on what you need. Teams that want AI traces linked to services and infrastructure often prefer one platform, while teams that need heavy evaluation workflows may add a dedicated tool.What is an AI agent operations platform?
A platform where AI agents carry out operations tasks such as incident diagnosis, inspection and capacity assessment on top of observability data.Is there a ready-made scorecard?
Yes. The GitHub repository ai-agent-observability-scorecard contains the full table and the proof-of-concept plan.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>tools</category>
    </item>
    <item>
      <title>Agentic AI Operations: Evidence-Carrying Handoffs for Multi-Agent Diagnosis</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Mon, 28 Sep 2026 09:57:18 +0000</pubDate>
      <link>https://dev.to/bonree-js/agentic-ai-operations-evidence-carrying-handoffs-for-multi-agent-diagnosis-1ko7</link>
      <guid>https://dev.to/bonree-js/agentic-ai-operations-evidence-carrying-handoffs-for-multi-agent-diagnosis-1ko7</guid>
      <description>&lt;p&gt;TL;DR — In a multi-agent operations workflow, the orchestrator's answer is only as trustworthy as the claims its sub-agents hand back. Structuring those handoffs as verifiable evidence, reasoning by elimination across the service topology, and replaying past incidents as a regression suite make multi-agent diagnosis easier to audit and improve.&lt;br&gt;
What is a multi-agent operations workbench? It is a system in which an orchestrating agent interprets an operator's request and routes parts of it to specialist agents, such as incident diagnosis, Q&amp;amp;A, reporting, or remediation.&lt;br&gt;
Written by the Bonree observability team. Bonree | Sage AI (Bonree ONE • Sage AI) is one such workbench: Bonree describes it as an AI Ops agent workbench that recognizes user intent through natural language and orchestrates the right agents, tools, and skills across unified operations and AI scenarios, home to a team of specialized agents with multi-agent collaboration for complex operational tasks. The patterns below are general design ideas, not descriptions of how any product is configured or operated.&lt;br&gt;
Keywords: multi-agent systems, LLM agents, agent reliability, agent trustworthiness, trustworthy AI, verifiable AI, root cause analysis, AIOps, AI observability, LLM observability, SRE, OpenTelemetry, DevOps&lt;br&gt;
Why handoffs are the weak point&lt;br&gt;
Discussions of multi-agent ops usually focus on routing. The harder problem is what comes back. If a specialist returns "the database looks unhealthy," the orchestrator can only repeat it, and a human reviewer cannot tell whether it came from a query, a runbook, or a guess.&lt;br&gt;
This matters more as agents are trusted with more autonomy. An orchestrator that acts on a specialist's claim without a way to check it is making a leap of faith, and that risk compounds across multi-step workflows: each additional hop is another place for an unverified claim to become the input to the next one. Evidence-carrying handoffs don't remove that risk, but they keep the chain auditable instead of just fast.&lt;br&gt;
Pattern 1: Evidence-carrying handoffs&lt;br&gt;
Ask each specialist to return a structured answer instead of prose. An illustrative structure (not a Bonree ONE data format):&lt;br&gt;
Field   Purpose&lt;br&gt;
Claim   The conclusion, in one sentence&lt;br&gt;
Confidence  Coarse (low / medium / high), not a fake-precise percentage&lt;br&gt;
Evidence    Where it came from: data source, query or reference, time range, entity, what was observed&lt;br&gt;
Ruled out   What was excluded, and which signal was checked&lt;br&gt;
Suggested next check    What to look at if the claim is wrong&lt;br&gt;
Evidence can point to OpenTelemetry traces, metrics, and logs, or any other re-runnable source.&lt;br&gt;
Pattern 2: Reason by elimination across the topology&lt;br&gt;
Checking every metric of every service is slow and noisy. Reasoning across the call chain is cheaper: if the entry service shows failures and every downstream service shows none, the fault most likely starts at the entry node, and whole downstream branches can be dropped in one step.&lt;br&gt;
Bonree has published a case of this shape from Bonree ONE • Sage AI's health analysis of a real service chain. The entry-point service showed 7 failed requests (a 0.04% error rate) while the two downstream services it called showed a zero error rate, so the analysis narrowed to the entry node instead of tracing further downstream. It separately flagged that all three services in the chain were deployed on the same host, which itself had 7 unresolved alerts, a structural single-point-of-failure risk that the per-service error rates alone did not show. Details, including the full topology reasoning and the resulting diagnostic report.&lt;br&gt;
Caveat: a zero error rate is not the same as healthy. Record which signal was checked, so latency or saturation problems are not silently excluded.&lt;br&gt;
Pattern 3: Share the tool layer, specialize the reasoning&lt;br&gt;
If every agent carries its own connectors, integration cost grows with the number of agents. Specialists work better when they differ in instructions and skills while models, tools, and knowledge come from one shared pool. Bonree ONE • Sage AI is described in the same terms: agents draw on a shared pool, so covering a new scenario does not require duplicating connections.&lt;br&gt;
Pattern 3a: Keep the shared layer framework-agnostic&lt;br&gt;
A related design question is which orchestration framework the specialists actually run on. If the shared tool and evidence layer from Pattern 3 is wired directly into one framework's APIs, every additional framework a team adopts means re-wiring those connectors again. A framework-agnostic tool contract avoids that: tool functions, schemas, and the evidence format from Pattern 1 are defined independently of any single orchestrator, with a thin adapter mapping them into whichever framework a given agent runs on.&lt;br&gt;
That contract still has to be observed consistently once it's running, and that's a separate problem: whatever framework a given specialist is built on, its tool calls, token usage, and evidence need to show up in the same shape in your traces. See the companion post on what differs across LangChain, LangGraph, Dify, and OpenClaw at the instrumentation layer, and why that's an AI Observability problem more than an orchestration one (Article 4 below).&lt;br&gt;
Pattern 4: Replay past incidents as a regression suite&lt;br&gt;
Keep closed incidents as fixtures: the alert, a topology snapshot, the time window, and the human-confirmed root cause. Replay them and track four things:&lt;br&gt;
Measure Question it answers&lt;br&gt;
Root-cause hit rate Did the final claim match the confirmed cause?&lt;br&gt;
Evidence validity   Do the cited queries reproduce what was claimed?&lt;br&gt;
Ruled-out precision Were excluded branches really innocent?&lt;br&gt;
Steps to first useful hypothesis    How quickly did the agents narrow the search?&lt;/p&gt;

&lt;p&gt;Limitations&lt;br&gt;
●Checking evidence proves a citation reproduces, not that the inference drawn from it is correct.&lt;br&gt;
●Re-running evidence needs stable, queryable tool interfaces.&lt;br&gt;
●Elimination reasoning is only as good as the topology it reads.&lt;br&gt;
●A replay suite built from a handful of incidents can overfit.&lt;br&gt;
●Framework-agnostic tool contracts add an adapter layer to maintain; for a single-framework team this is overhead until a second framework actually shows up.&lt;br&gt;
FAQ&lt;br&gt;
What is an evidence-carrying handoff? A structured sub-agent answer that includes its claim, the sources behind it, and what it ruled out, so an orchestrator or a human can verify it.&lt;br&gt;
Does multi-agent diagnosis remove human review? No. Evidence makes review faster and more specific.&lt;br&gt;
Does supporting multiple orchestration frameworks weaken the shared tool layer? Not if the tool and evidence contracts are defined independently of any framework; the framework only decides how an agent is orchestrated, not what evidence it must produce.&lt;br&gt;
Further reading&lt;br&gt;
●&lt;a href="https://en.bonree.com/en/sageai" rel="noopener noreferrer"&gt;https://en.bonree.com/en/sageai &lt;/a&gt;&lt;br&gt;
●&lt;a href="https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1"&gt;https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1&lt;/a&gt; &lt;br&gt;
●&lt;a href="https://en.bonree.com" rel="noopener noreferrer"&gt;https://en.bonree.com&lt;/a&gt; &lt;/p&gt;

&lt;h1&gt;
  
  
  observability #opentelemetry #devops #Bonree
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>llm</category>
    </item>
    <item>
      <title>Instrumenting LangChain, LangGraph, Dify, and OpenClaw: What Actually Differs at the Tracing Layer</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Mon, 28 Sep 2026 09:37:03 +0000</pubDate>
      <link>https://dev.to/bonree-js/instrumenting-langchain-langgraph-dify-and-openclaw-what-actually-differs-at-the-tracing-layer-jn0</link>
      <guid>https://dev.to/bonree-js/instrumenting-langchain-langgraph-dify-and-openclaw-what-actually-differs-at-the-tracing-layer-jn0</guid>
      <description>&lt;p&gt;The previous post walked through instrumenting an LLM call by hand: a span per model call, a histogram for token usage, a small set of fixed-cardinality labels. That works when there's one call site to wrap. It gets harder once a platform team is supporting several product teams that each picked a different agent framework, because LangChain, LangGraph, Dify, and OpenClaw don't expose the same thing to instrument in the same place.&lt;br&gt;
Disclosure: I work at Bonree, an observability vendor. Most of this post is vendor-neutral; Bonree ONE appears near the end.&lt;br&gt;
Four frameworks, four different instrumentation surfaces&lt;br&gt;
●LangChain runs in-process as a Python or JS library, and exposes a callback interface (handlers for on_llm_start, on_llm_end, on_tool_start, and similar events). Instrumentation attaches to these callbacks directly in the calling application's process.&lt;br&gt;
●LangGraph adds an explicit state machine on top: nodes, edges, and checkpoints. The natural place to attach a span is per node execution and per state transition, not per model call, since a single node can make zero, one, or several model calls.&lt;br&gt;
●Dify is mostly consumed as a hosted, low-code platform: a large part of the pipeline runs inside Dify's own engine rather than in the calling application's process, so in-process callbacks aren't available for most of it. Visibility typically comes from Dify's own execution logs or webhooks instead.&lt;br&gt;
●OpenClaw is a standalone, always-on agent runtime rather than a library you call, so instrumentation has to happen at the boundary between it and everything else, its API or webhook surface, instead of inline in application code.&lt;br&gt;
In practice, this means "add tracing" means four different things depending on the framework: wrap a callback, wrap a node function, consume an execution log, or hook a webhook. None of that is difficult in isolation. It becomes real, ongoing work when a platform team is the one expected to keep all four producing comparable data.&lt;br&gt;
Why comparable matters more than complete&lt;br&gt;
It's not enough for each framework to be traced somehow. If LangChain traces carry gen_ai.agent.name and LangGraph traces carry graph.node.name with no equivalent, per-agent cost and error-rate comparisons across frameworks silently break. The fix is the same one from the previous post: standardize on a shared attribute schema, such as OpenTelemetry's GenAI semantic conventions, and map every framework's own event model onto it, rather than letting each framework's instrumentation invent its own field names.&lt;br&gt;
What automatic instrumentation is actually buying you&lt;br&gt;
Given that mapping work, "out-of-the-box support for framework X" in an observability tool is worth more than it sounds like on a feature list. It's not just a connector, it's someone else having already done the work of mapping LangChain's callbacks, LangGraph's node/edge model, Dify's execution logs, and OpenClaw's API surface onto one consistent schema, so a platform team isn't building and maintaining four separate adapters by hand.&lt;br&gt;
Limitations&lt;br&gt;
●All four frameworks change quickly. Automatic instrumentation still needs to be re-validated against new framework versions, the same way hand-written instrumentation does.&lt;br&gt;
●A hosted platform like Dify may not expose every internal step to any external collector, in-process or not, so some signals may only ever be as detailed as what the platform's own logs provide.&lt;br&gt;
●Consistent collection solves comparability, not governance. Turning collected data into per-team cost and usage rollups is the labeling and aggregation problem covered in the previous post, not something instrumentation coverage does by itself.&lt;br&gt;
Where Bonree ONE fits&lt;br&gt;
Bonree ONE 4.0's AI Observability capability instruments Python, Node.js, and Java applications, with automatic adaptation for common model-native APIs and out-of-the-box support for LangChain, LangGraph, Dify, and OpenClaw, among other frameworks. See &lt;a href="https://en.bonree.com/en/aIObservability" rel="noopener noreferrer"&gt;https://en.bonree.com/en/aIObservability&lt;/a&gt;  and the post &lt;a href="https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1"&gt;https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1&lt;/a&gt;  for the underlying architecture.&lt;br&gt;
A question for readers&lt;br&gt;
If you're running agents on more than one of these frameworks, how are you keeping their telemetry comparable today: a shared schema you enforce yourselves, a vendor's auto-instrumentation, or have you not tried to reconcile them yet?&lt;/p&gt;

&lt;h1&gt;
  
  
  observability #opentelemetry #devops #Bonree
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>opentelemetry</category>
    </item>
    <item>
      <title>Why LLM Agent Token Costs Are Hard to Attribute, and What Telemetry Design Can Do About It</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Thu, 24 Sep 2026 08:53:49 +0000</pubDate>
      <link>https://dev.to/bonree-js/why-llm-agent-token-costs-are-hard-to-attribute-and-what-telemetry-design-can-do-about-it-iil</link>
      <guid>https://dev.to/bonree-js/why-llm-agent-token-costs-are-hard-to-attribute-and-what-telemetry-design-can-do-about-it-iil</guid>
      <description>&lt;p&gt;AI observability is the practice of collecting traces, metrics, and logs from LLM-backed applications so that latency, quality, and token cost can be explained per model call, per agent, and per conversation, not only per HTTP request.&lt;br&gt;
Disclosure: I work at Bonree, an observability vendor. Most of this post is vendor-neutral; Bonree ONE appears near the end.&lt;br&gt;
The invoice tells you the total, not the cause&lt;br&gt;
One user message to an agent can trigger several model calls, tool calls, and retries. Provider dashboards often aggregate by API key or project, which does not show which agent, which model choice, which team, or which conversation drove the spend. That gap is what telemetry design has to close.&lt;br&gt;
Design principles&lt;br&gt;
●Measure per model call, not per request. A request-level number hides which step in the chain was expensive.&lt;br&gt;
●Split by cardinality. Use metrics for totals, with a small fixed set of labels such as model, agent, team, and token type. Keep unbounded identifiers, such as conversation or user IDs, in traces, where per-conversation detail belongs. An ID used as a metric label creates a new time series for every ID.&lt;br&gt;
●Store token counts, convert to currency later. Prices change. Raw counts stay valid, and a price table applied at analysis time keeps history correct.&lt;br&gt;
●Treat prompt and completion capture as a separate decision. Content often contains customer data and needs its own review.&lt;br&gt;
●Know how your provider counts tokens. Cached and reasoning tokens may be reported differently across providers, so check what each field includes before comparing numbers.&lt;br&gt;
●Make spend attributable to a team, not only a model. Once agent, model, and token-type labels exist on the same metrics, a team-level rollup is a query, not a separate reporting pipeline, and it turns a cost question into a governance one: which teams are running agents, at what volume, and with what effect.&lt;br&gt;
Use a standard path&lt;br&gt;
OpenTelemetry's GenAI semantic conventions define shared names for LLM and agent telemetry. They are still evolving, so pin versions and re-check on upgrade. Keeping collection vendor-neutral also turns the choice of backend into a configuration decision rather than an instrumentation rewrite, which matters to any DevOps or SRE team that expects to change tools someday.&lt;br&gt;
A minimal example&lt;br&gt;
The snippet below is vendor-neutral OpenTelemetry code that illustrates the metrics-versus-traces split. It is not a Bonree ONE configuration. Tested with opentelemetry-sdk 1.44.0 and opentelemetry-semantic-conventions 0.65b0; the GenAI conventions are still evolving, so re-check attribute names when you upgrade.&lt;br&gt;
python&lt;/p&gt;

&lt;p&gt;from opentelemetry import trace, metrics&lt;/p&gt;

&lt;h1&gt;
  
  
  Assumes a TracerProvider and MeterProvider are configured elsewhere.
&lt;/h1&gt;

&lt;p&gt;tracer = trace.get_tracer("agent.instrumentation")&lt;br&gt;
meter = metrics.get_meter("agent.instrumentation")&lt;/p&gt;

&lt;p&gt;token_usage = meter.create_histogram(&lt;br&gt;
    "gen_ai.client.token.usage", unit="{token}",&lt;br&gt;
    description="tokens per call",&lt;br&gt;
)&lt;/p&gt;

&lt;p&gt;def call_llm(client, *, model, messages, agent, team, session_id):&lt;br&gt;
    with tracer.start_as_current_span(f"chat {model}") as span:&lt;br&gt;
        span.set_attribute("gen_ai.operation.name", "chat")&lt;br&gt;
        span.set_attribute("gen_ai.request.model", model)&lt;br&gt;
        span.set_attribute("gen_ai.agent.name", agent)&lt;br&gt;
        span.set_attribute("gen_ai.conversation.id", session_id)  # span only, never a metric label&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    resp = client.chat(model=model, messages=messages)  # stand-in for your SDK call
    span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)

    labels = {"gen_ai.request.model": model, "gen_ai.agent.name": agent, "team": team}
    token_usage.record(resp.usage.input_tokens, {**labels, "gen_ai.token.type": "input"})
    token_usage.record(resp.usage.output_tokens, {**labels, "gen_ai.token.type": "output"})
    return resp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Note that the conversation ID lives on the span only, while team is a small, fixed-cardinality label, so it is safe to put on the metric. In a quick local test with a stub client and two conversations across two teams, the metric produced four series (two teams × input/output), not one per conversation.&lt;br&gt;
Where Bonree ONE fits&lt;br&gt;
Bonree ONE 4.0 includes an AI Observability capability that Bonree describes as providing end-to-end tracing, span-level analysis, token usage monitoring, performance metrics, and session-level context tracking for large-model and agent environments. Bonree ONE also supports standard OpenTelemetry ingestion of traces, metrics, and logs. Whether a team-level rollup like the one above is useful on top of that depends on whether agent and team labels are already being recorded consistently, which is the instrumentation-design question this post is about, not something any specific platform does automatically. See &lt;a href="https://en.bonree.com/en/aIObservability" rel="noopener noreferrer"&gt;&lt;/a&gt;and &lt;a href="https://en.bonree.com/en/opentelemetrystandard" rel="noopener noreferrer"&gt;https://en.bonree.com/en/opentelemetrystandard&lt;/a&gt; , or the post &lt;a href="https://en.bonree.com/en/blog/aiobservabilityinpracticeinstrumentingagentchainsnotjustapicalls" rel="noopener noreferrer"&gt;https://en.bonree.com/en/blog/aiobservabilityinpracticeinstrumentingagentchainsnotjustapicalls&lt;/a&gt; . More at &lt;a href="https://en.bonree.com" rel="noopener noreferrer"&gt;https://en.bonree.com&lt;/a&gt; .&lt;br&gt;
A question for readers&lt;br&gt;
How does your team attribute agent token spend today: per API key, per agent, per team, or per conversation?&lt;/p&gt;

&lt;h1&gt;
  
  
  observability #opentelemetry #devops #Bonree
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>devops</category>
      <category>llm</category>
    </item>
    <item>
      <title>Building AI Observability for the Native Stack: Architecture Design and Engineering Practice from Bonree ONE 4.0</title>
      <dc:creator>Bonree-Js</dc:creator>
      <pubDate>Mon, 14 Sep 2026 07:10:09 +0000</pubDate>
      <link>https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1</link>
      <guid>https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1</guid>
      <description>&lt;p&gt;AI is changing observability and operations platforms in two directions at once, and it's easy to conflate them. The first direction is familiar: AI applications — RAG pipelines, LLM-backed customer service agents, coding copilots — are now part of the production stack, and they behave in ways traditional APM was never designed to see. The second direction is less discussed but arguably more consequential: AI is no longer only the thing being monitored. It's increasingly the thing doing the monitoring, and the thing taking action on what it finds.&lt;/p&gt;

&lt;p&gt;This article walks through how we approached both problems in Bonree ONE 4.0, the latest release of our observability and AIOps platform. Rather than a feature tour, the goal here is to explain the reasoning behind three capability areas we built — AI Observability, an AI-driven natural-language query layer called SmartAsk, and an autonomous agent workbench called Bonree ONE·Sage AI — and the production challenges that shaped their architecture.&lt;/p&gt;

&lt;p&gt;Six problems that don't show up on a status page&lt;/p&gt;

&lt;p&gt;A quick example sets the tone for all of them. An insurance customer of ours ran an AI phone agent for customer service. A caller asked how to purchase a policy; the agent misparsed the request and responded with instructions for cancelling one instead. No exception was thrown, no error rate moved, no health check failed — by every conventional signal, the call was a complete success. Hallucination and answer quality are orthogonal to uptime.&lt;/p&gt;

&lt;p&gt;That example generalizes into six patterns we kept running into, all of which shaped the architecture below:&lt;/p&gt;

&lt;p&gt;AI applications fail silently — confidently wrong output that no uptime or latency metric ever flags.&lt;br&gt;
Agent output gets trusted without being understood — code or decisions an operator can't debug when they eventually break.&lt;br&gt;
Bigger context windows don't remove the need for data governance — we've seen our own tooling get visibly worse at intent recognition once a session accumulates enough unfiltered context, well before hitting a hard token limit.&lt;br&gt;
Autonomy raises the security bar — an agent acts at machine speed, and skills that are individually safe can combine into risks nobody reviewed for.&lt;br&gt;
Cost doesn't fail gracefully — a single agentic query can burn hundreds of thousands of tokens, and unlike a slow response, an overrun is invisible until someone reads the invoice.&lt;br&gt;
Org structures haven't caught up — "AI does everything" scales to a team of one, not a team of hundreds, and how a human and an agent actually share an incident war room is still being worked out in practice.&lt;/p&gt;

&lt;p&gt;The rest of this article is about the architecture that came out of treating these as engineering problems rather than caveats: three capability areas in Bonree ONE 4.0 — AI Observability, a natural-language query layer called SmartAsk, and an autonomous agent workbench called Bonree ONE·Sage AI.&lt;/p&gt;

&lt;p&gt;AI Observability: treating AI applications as first-class citizens in the trace store&lt;/p&gt;

&lt;p&gt;The instinct when a new class of application shows up is to bolt a metric onto an existing dashboard. That doesn't hold up here, because the unit of work itself has changed. A single user request into a RAG or multi-agent application can fan out into a dozen internal steps — retrieval, several sequential or parallel model calls, tool invocations, result synthesis — each with its own latency, token cost, and independent chance of failure. Collapsing all of that into one opaque "request," the way a conventional APM trace would, throws away exactly the information needed to debug it.&lt;/p&gt;

&lt;p&gt;Show Image&lt;/p&gt;

&lt;p&gt;Collection. Instrumentation covers Python, Node.js, and Java, with automatic adaptation for common model-native APIs and out-of-the-box support for LangChain, LangGraph, Dify, and OpenClaw, among other agent frameworks. It's non-invasive — for a framework that's already supported, there's no code change required to start collecting data, which matters more than it might sound: requiring every team to hand-instrument their LangGraph pipeline before they get any observability is a real adoption barrier, not a minor inconvenience. Collection also speaks OpenTelemetry natively — Traces, Metrics, and Logs travel over OTLP — so AI telemetry isn't a second, proprietary data path sitting next to whatever teams already run for the rest of their stack.&lt;/p&gt;

&lt;p&gt;Redaction is handled at collection time, not after storage. Prompts and completions routinely contain customer data, and capturing full input/output at every span means you've effectively built a PII pipeline unless sensitive content is filtered before it lands anywhere persistent. The platform supports configurable masking rules and automatic PII identification and filtering applied at the point of capture.&lt;/p&gt;

&lt;p&gt;Processing pipeline. Once collected, data moves through real-time desensitization and format normalization, then metric extraction (performance indicators land in structured storage), then an analysis layer that runs hallucination detection and answer-quality scoring. This last stage is what turns raw traces into something closer to a quality signal rather than just a performance one — necessary precisely because, as the insurance example showed, quality problems don't otherwise register as failures at all.&lt;/p&gt;

&lt;p&gt;What you actually see. The platform exposes this data through several coordinated views rather than one dashboard trying to do everything:&lt;/p&gt;

&lt;p&gt;An application overview lists every AI service with request volume, error rate, response time, and total token consumption, so an anomalous application is visible at a glance before you drill into anything.&lt;br&gt;
Call chain analysis lists individual AI invocations with full input/output content, response time, and status, filterable by application, trace ID, user ID, or session ID.&lt;br&gt;
Call chain detail is where debugging actually happens: a Call Tree renders the trace as a time-series Gantt chart, with an "ALL" mode showing every span type (HTTP, chain, prompt, llm, parser) and an "LLM" mode that filters down to just the model-related nodes for focused analysis of the reasoning layer. A companion Call Map renders the same trace as a topology graph, with each node showing average response time, request count, and token count. Clicking any node opens a detail panel with four tabs — the input/output content and attributes, a timing breakdown, the code stack that triggered the span, and any errors or logs — which is usually enough to go from "this was slow" to a specific, fixable cause, such as a system prompt that had grown past several thousand tokens after multiple rounds of tool output were appended to context.&lt;br&gt;
A performance view aggregates response time, request count, error count, and model request count with trend comparisons against the prior day.&lt;br&gt;
A token view breaks down total consumption by model, separating input and output tokens, with trend charts over time — the level of granularity needed to answer "which model or which application is actually driving this month's bill," rather than only a top-line total.&lt;br&gt;
A model view aggregates by model rather than by application, so teams running more than one LLM in production can compare call volume, average latency, error rate, and token cost side by side and make an informed choice about which model earns its cost for a given task.&lt;br&gt;
A session view is arguably the most important one for conversational and agentic systems specifically, because the most common quality failure — degradation as context accumulates — is invisible if you only ever look at individual requests. Session analysis aggregates every trace belonging to one multi-turn conversation: total token consumption, trace count, and duration at the list level; a per-trace breakdown showing input preview, response time, and LLM/tool call counts inside a session; a waterfall view of any individual trace's internal span hierarchy, color-coded by node type (Agent, Chain, LLM, Task); and — distinctly useful for multi-agent systems — an Agent collaboration topology that shows, round by round, which node in a multi-agent execution consumed how much time and how many tokens. That last view is what makes it possible to say "the second agent in the chain is where both the latency and the token spend are concentrated" instead of only knowing that a three-hour session was expensive without knowing why.&lt;/p&gt;

&lt;p&gt;Alerts integrate directly into the same application detail view, categorized by severity (fatal, critical, warning, general, reminder) with filtering by status and rule type, so a spike surfaced in AI Observability doesn't require switching to a separate alerting tool to investigate.&lt;/p&gt;

&lt;p&gt;SmartAsk: making the data conversational without giving up rigor&lt;/p&gt;

&lt;p&gt;AI Observability answers "what is happening inside my AI applications." SmartAsk answers a related but different question: "how do I get an answer out of all my observability data — AI-related or not — without writing a query." It's a natural-language interface positioned as an expert-level Q&amp;amp;A layer over the whole platform, not just the AI-observability data described above.&lt;/p&gt;

&lt;p&gt;The core technical claim here is "deep data understanding": no data modeling or field mapping step is required before you can ask a question. The system automatically resolves the semantics of metrics, logs, traces, and events, and infers the right data source, filter conditions, and time range from the question itself, rather than requiring the user to specify them explicitly the way a dashboard query builder would. In practice this means the barrier to getting an answer drops from "know the metric name, the right PromQL syntax, and which dashboard has it" to "describe what you want to know."&lt;/p&gt;

&lt;p&gt;Rather than starting from a blank prompt every time, the platform ships with more than 30 pre-built scenarios across six categories — health inspection, fault diagnosis, performance optimization, change evaluation, ops governance, and cross-cutting "fusion" scenarios that combine several data types in one answer. These aren't templates in the sense of fixed report formats; they're curated starting points distilled from common patterns across customer deployments, meant to be used as-is or adapted with different targets, time ranges, or thresholds.&lt;/p&gt;

&lt;p&gt;Answers come back in a consistent three-part structure: an AI-generated summary in plain language, the underlying raw data, and a visualization — so a reader can trust the headline conclusion, verify it against the actual numbers, or hand the chart to someone else without redoing the analysis. Conversations persist as history that can be continued with follow-up questions, and a query that turns out to be useful more than once can be saved and reused rather than re-typed. Results export to PDF or DOC directly, and — a detail that matters operationally — a one-off query can be promoted into a permanent dashboard widget with one action, so an ad hoc investigation and a recurring monitoring need aren't two different workflows requiring two different tools.&lt;/p&gt;

&lt;p&gt;The intended audience is deliberately broad: not just the on-call engineer who already knows the query language, but developers and business stakeholders who need an answer from operational data but have never written PromQL or SQL and shouldn't need to.&lt;/p&gt;

&lt;p&gt;Bonree ONE·Sage AI: from answering questions to taking action&lt;/p&gt;

&lt;p&gt;SmartAsk gets you an answer. Bonree ONE·Sage AI is built to go further — from a natural-language description of a problem or task to actually carrying it out, using models, tools, knowledge bases, skills, and pre-built or custom agents behind a single conversational interface. The framing we use internally is that this is meant to be the difference between AI as a tool you operate and AI as a colleague that operates alongside you — a distinction that sounds like marketing language until you look at what it requires architecturally, which is considerably more than a chat UI in front of an LLM.&lt;/p&gt;

&lt;p&gt;Layered architecture. Bonree ONE·Sage AI is built as seven layers, each addressing a distinct concern:&lt;/p&gt;

&lt;p&gt;Show Image&lt;/p&gt;

&lt;p&gt;Data source layer — observability signals (logs, metrics, traces, alerts, events), operational assets (CMDB, runbooks/knowledge bases, ITSM tickets), and file assets (scripts, environment variables, credentials). All three categories are treated as inputs an agent might need, not just telemetry.&lt;br&gt;
Connector layer — the Model Context Protocol (MCP) as the standard channel for agent-to-tool and agent-to-system communication, connecting to CMDB and ticketing systems; a CLI layer that does pre-processing before data reaches the model, specifically to reduce how many tokens MCP calls consume — a detail worth noting because it's a direct response to the cost problem described earlier, not a generic engineering nicety; plus conventional API and script access.&lt;br&gt;
Security review layer — every signal and asset entering the system goes through specification validation, permission checks, and a security review before it's usable, and this is deliberately the first line of defense, not a step bolted on after something is already running.&lt;br&gt;
Security sandbox layer — this is the layer that makes autonomous execution tolerable in a production environment. It provides resource, network, filesystem, process, and multi-tenant isolation for every agent execution. Because a complex task involving multiple collaborating agents can legitimately run for minutes or, in some cases, hours, isolation has to hold for the duration of a long-running, potentially unattended job, not just for a single quick call. On top of isolation, there's ingress filtering that blocks malicious instructions before they reach an agent, egress filtering that strips sensitive information from output, execution permission controls with a human-approval gate for consequential actions, and a requirement that agent and skill creation and publishing go through supervisor review before they're available to be invoked at all.&lt;br&gt;
Scheduling and orchestration layer — managed across four dimensions: reliability (fallback strategies, error retry, circuit breaking so one flaky tool call doesn't cascade into a stuck task), validity (schema validation to keep malformed data from silently propagating between steps), agility (a primary agent that interprets intent, decomposes a task, and routes sub-tasks to the right sub-agent, monitoring and correcting course during execution), and coherence (long- and short-term memory management, including memory decay and update mechanisms, so an agent adapts to a user's patterns over time without letting stale context degrade later reasoning — the same accumulation problem described in the AI Observability section, addressed architecturally here rather than just observed).&lt;br&gt;
Interaction control layer — web UI, multi-channel access, alert-triggered execution, scheduled/periodic tasks, and session management, covering both interactive and unattended invocation.&lt;br&gt;
Application scenario layer — the top-level entry points: development and testing, release and change management, inspection and maintenance, emergency recovery, disaster-recovery drills, and ops governance.&lt;/p&gt;

&lt;p&gt;Two orchestration modes, chosen per task, not tuned on a dial. For standardized, well-understood work — a routine inspection, a change-approval flow — a fixed workflow is the right tool: explicit steps, predictable execution order, straightforward to audit against, and it fails in enumerable ways. For open-ended fault diagnosis, where the root cause is genuinely unknown at the start and the next useful action depends entirely on what the previous one revealed, a scripted workflow breaks down almost immediately, because you can't pre-write a decision tree for a failure mode you haven't seen yet. That calls for the autonomous-decision mode, where the agent reasons step by step and decides what to check next based on what it just found.&lt;/p&gt;

&lt;p&gt;We've watched this second mode play out in a real diagnostic session, and it's a useful illustration of how the security sandbox and orchestration layers work together rather than as separate concerns. An agent was asked to investigate a Java service running in a container and attempt recovery if warranted, with no information given up front about whether the environment was Kubernetes or plain Docker, or where the container lived. It probed the environment, determined it was Docker, and located the target container, then worked through memory, thread, CPU, garbage-collection, and log analysis autonomously — the kind of multi-step investigation that would otherwise require an engineer to log in and do by hand. At two specific points, where the next command would have read deep internal container state, it stopped, displayed the exact command it intended to run, and waited for explicit human confirmation before proceeding. It didn't ask permission for read-only statistics gathering. It asked only for the operations classified as having genuine potential impact — a concrete instance of the "execution permission control + human approval gate" mechanism described in the sandbox layer above, not a one-off safety feature bolted on for that specific session.&lt;/p&gt;

&lt;p&gt;Building agents, not just using them. The workbench separates three roles — administrators handle model access and platform-wide security and audit configuration; creators build skills and agents in a dedicated workspace, using either the workflow or autonomous-decision construction method, then publish for their own use or submit for review to share more broadly; users consume whatever has been published, either the built-in library or anything added from an internal resource marketplace. The same person is commonly all three at different times — building a skill in the morning and consuming someone else's in the afternoon — and the platform is built around that overlap rather than assuming rigid role separation.&lt;/p&gt;

&lt;p&gt;Out of the box, the platform includes more than 40 MCP-based tools and is compatible with external MCP servers, more than 10 ready-to-use skills including a "deep service diagnostics" skill, and a knowledge base that accepts standard document formats (Markdown, TXT, PDF) as well as API-based ingestion, so an organization's own runbooks and incident playbooks — the kind of institutional knowledge that otherwise lives in a senior engineer's head — can be folded directly into what an agent draws on. Several pre-built expert agents ship with the platform, including a terminal/host diagnostics agent and a database-specialist agent built on patterns distilled from experienced DBA troubleshooting workflows, usable directly or scheduled to run as an unattended, always-on specialist.&lt;/p&gt;

&lt;p&gt;Why this is worth the architectural complexity. The value case breaks into four tiers that build on each other. The most direct is efficiency: routine inspection, troubleshooting, reporting, and alert handling can run end-to-end through an agent, freeing operators from repetitive work and materially shortening mean time to resolution once a team can pull full-stack data with one query instead of stitching it together by hand. The second is an asset value that's easy to undervalue until you've lost it: a senior engineer's troubleshooting instinct, previously undocumented and lost when they leave or change roles, gets encoded as a reusable skill or knowledge-base entry instead — turning tacit, personal know-how into a structured, transferable asset. The third is a shift in operating model, from reactive to anticipatory: scheduled inspection combined with baseline analysis surfaces risk before it becomes an incident, and direct integration with ticketing and CMDB systems closes the loop from detection through resolution rather than stopping at notification. The fourth is more strategic than operational: full execution logging supports the audit requirements that regulated industries — finance, government — actually need, which is a precondition for deploying autonomous agents in those environments at all, not an optional nice-to-have.&lt;/p&gt;

&lt;p&gt;What's still unresolved&lt;/p&gt;

&lt;p&gt;It would be dishonest to present all of this as a solved problem, and a few things are worth naming directly.&lt;/p&gt;

&lt;p&gt;Skill-level security review is necessary but not sufficient. A skill that queries a monitoring API and a skill that restarts a process can each look completely safe in isolation and still combine into something neither reviewer anticipated once an agent chains them together at runtime in a sequence nobody explicitly tested. Static, pre-publish review catches the safety of individual components; it does not catch emergent risk from composition. The mitigations available so far — runtime ingress/egress filtering that doesn't depend on having anticipated the specific dangerous combination in advance, and treating the human-confirmation gate as a backstop rather than a complete answer — are partial, not complete, and we don't think this is a solved problem industry-wide.&lt;/p&gt;

&lt;p&gt;Cost governance is a genuine, ongoing tension rather than a one-time optimization. The internal example cited earlier — token spend growing substantially even as the explicit goal of adopting AI tooling was cost reduction — isn't unusual, and treating token consumption as an engineering-visible signal (per model, per session, per conversation turn, as described in the AI Observability section) is necessary precisely because that tension doesn't resolve itself; it has to be actively monitored and traded off against agent autonomy on purpose.&lt;/p&gt;

&lt;p&gt;And organizationally, the question of how a human on-call engineer and an autonomous diagnostic agent actually collaborate during a live incident — who has authority to decide, how escalation works when the agent's confidence is low, what a shared "war room" looks like when one participant is a model — is still being worked out in practice at most organizations we've talked to, ourselves included. Tooling can support that collaboration, but it can't yet fully define it.&lt;/p&gt;

&lt;p&gt;Where this leaves the three pieces&lt;/p&gt;

&lt;p&gt;AI Observability, SmartAsk, and Bonree ONE·Sage AI aren't three independent features so much as three layers of the same problem. Observability tells you what's actually happening — including inside the AI applications that used to be a blind spot entirely. SmartAsk makes that information reachable in natural language, without requiring every stakeholder to learn a query syntax first. And Bonree ONE·Sage AI is where understanding turns into action, under a governance model — layered isolation, staged review, and a confirmation gate scoped specifically to consequential operations — built for the fact that the system doing the acting is no longer only a human being.&lt;/p&gt;

&lt;p&gt;None of this makes AI-native operations a solved problem. But it's a different, and more specific, problem than "add a chatbot to the dashboard," and the architecture ends up looking correspondingly different once you take that seriously.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
