DEV Community: Bonree-Js

Agentic AI Operations: Evidence-Carrying Handoffs for Multi-Agent Diagnosis

Bonree-Js — Mon, 28 Sep 2026 09:57:18 +0000

TL;DR — In a multi-agent operations workflow, the orchestrator's answer is only as trustworthy as the claims its sub-agents hand back. Structuring those handoffs as verifiable evidence, reasoning by elimination across the service topology, and replaying past incidents as a regression suite make multi-agent diagnosis easier to audit and improve.
What is a multi-agent operations workbench? It is a system in which an orchestrating agent interprets an operator's request and routes parts of it to specialist agents, such as incident diagnosis, Q&A, reporting, or remediation.
Written by the Bonree observability team. Bonree | Sage AI (Bonree ONE • Sage AI) is one such workbench: Bonree describes it as an AI Ops agent workbench that recognizes user intent through natural language and orchestrates the right agents, tools, and skills across unified operations and AI scenarios, home to a team of specialized agents with multi-agent collaboration for complex operational tasks. The patterns below are general design ideas, not descriptions of how any product is configured or operated.
Keywords: multi-agent systems, LLM agents, agent reliability, agent trustworthiness, trustworthy AI, verifiable AI, root cause analysis, AIOps, AI observability, LLM observability, SRE, OpenTelemetry, DevOps
Why handoffs are the weak point
Discussions of multi-agent ops usually focus on routing. The harder problem is what comes back. If a specialist returns "the database looks unhealthy," the orchestrator can only repeat it, and a human reviewer cannot tell whether it came from a query, a runbook, or a guess.
This matters more as agents are trusted with more autonomy. An orchestrator that acts on a specialist's claim without a way to check it is making a leap of faith, and that risk compounds across multi-step workflows: each additional hop is another place for an unverified claim to become the input to the next one. Evidence-carrying handoffs don't remove that risk, but they keep the chain auditable instead of just fast.
Pattern 1: Evidence-carrying handoffs
Ask each specialist to return a structured answer instead of prose. An illustrative structure (not a Bonree ONE data format):
Field Purpose
Claim The conclusion, in one sentence
Confidence Coarse (low / medium / high), not a fake-precise percentage
Evidence Where it came from: data source, query or reference, time range, entity, what was observed
Ruled out What was excluded, and which signal was checked
Suggested next check What to look at if the claim is wrong
Evidence can point to OpenTelemetry traces, metrics, and logs, or any other re-runnable source.
Pattern 2: Reason by elimination across the topology
Checking every metric of every service is slow and noisy. Reasoning across the call chain is cheaper: if the entry service shows failures and every downstream service shows none, the fault most likely starts at the entry node, and whole downstream branches can be dropped in one step.
Bonree has published a case of this shape from Bonree ONE • Sage AI's health analysis of a real service chain. The entry-point service showed 7 failed requests (a 0.04% error rate) while the two downstream services it called showed a zero error rate, so the analysis narrowed to the entry node instead of tracing further downstream. It separately flagged that all three services in the chain were deployed on the same host, which itself had 7 unresolved alerts, a structural single-point-of-failure risk that the per-service error rates alone did not show. Details, including the full topology reasoning and the resulting diagnostic report.
Caveat: a zero error rate is not the same as healthy. Record which signal was checked, so latency or saturation problems are not silently excluded.
Pattern 3: Share the tool layer, specialize the reasoning
If every agent carries its own connectors, integration cost grows with the number of agents. Specialists work better when they differ in instructions and skills while models, tools, and knowledge come from one shared pool. Bonree ONE • Sage AI is described in the same terms: agents draw on a shared pool, so covering a new scenario does not require duplicating connections.
Pattern 3a: Keep the shared layer framework-agnostic
A related design question is which orchestration framework the specialists actually run on. If the shared tool and evidence layer from Pattern 3 is wired directly into one framework's APIs, every additional framework a team adopts means re-wiring those connectors again. A framework-agnostic tool contract avoids that: tool functions, schemas, and the evidence format from Pattern 1 are defined independently of any single orchestrator, with a thin adapter mapping them into whichever framework a given agent runs on.
That contract still has to be observed consistently once it's running, and that's a separate problem: whatever framework a given specialist is built on, its tool calls, token usage, and evidence need to show up in the same shape in your traces. See the companion post on what differs across LangChain, LangGraph, Dify, and OpenClaw at the instrumentation layer, and why that's an AI Observability problem more than an orchestration one (Article 4 below).
Pattern 4: Replay past incidents as a regression suite
Keep closed incidents as fixtures: the alert, a topology snapshot, the time window, and the human-confirmed root cause. Replay them and track four things:
Measure Question it answers
Root-cause hit rate Did the final claim match the confirmed cause?
Evidence validity Do the cited queries reproduce what was claimed?
Ruled-out precision Were excluded branches really innocent?
Steps to first useful hypothesis How quickly did the agents narrow the search?

Limitations
●Checking evidence proves a citation reproduces, not that the inference drawn from it is correct.
●Re-running evidence needs stable, queryable tool interfaces.
●Elimination reasoning is only as good as the topology it reads.
●A replay suite built from a handful of incidents can overfit.
●Framework-agnostic tool contracts add an adapter layer to maintain; for a single-framework team this is overhead until a second framework actually shows up.
FAQ
What is an evidence-carrying handoff? A structured sub-agent answer that includes its claim, the sources behind it, and what it ruled out, so an orchestrator or a human can verify it.
Does multi-agent diagnosis remove human review? No. Evidence makes review faster and more specific.
Does supporting multiple orchestration frameworks weaken the shared tool layer? Not if the tool and evidence contracts are defined independently of any framework; the framework only decides how an agent is orchestrated, not what evidence it must produce.
Further reading
●https://en.bonree.com/en/sageai
●https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1
●https://en.bonree.com

observability #opentelemetry #devops #Bonree

Instrumenting LangChain, LangGraph, Dify, and OpenClaw: What Actually Differs at the Tracing Layer

Bonree-Js — Mon, 28 Sep 2026 09:37:03 +0000

The previous post walked through instrumenting an LLM call by hand: a span per model call, a histogram for token usage, a small set of fixed-cardinality labels. That works when there's one call site to wrap. It gets harder once a platform team is supporting several product teams that each picked a different agent framework, because LangChain, LangGraph, Dify, and OpenClaw don't expose the same thing to instrument in the same place.
Disclosure: I work at Bonree, an observability vendor. Most of this post is vendor-neutral; Bonree ONE appears near the end.
Four frameworks, four different instrumentation surfaces
●LangChain runs in-process as a Python or JS library, and exposes a callback interface (handlers for on_llm_start, on_llm_end, on_tool_start, and similar events). Instrumentation attaches to these callbacks directly in the calling application's process.
●LangGraph adds an explicit state machine on top: nodes, edges, and checkpoints. The natural place to attach a span is per node execution and per state transition, not per model call, since a single node can make zero, one, or several model calls.
●Dify is mostly consumed as a hosted, low-code platform: a large part of the pipeline runs inside Dify's own engine rather than in the calling application's process, so in-process callbacks aren't available for most of it. Visibility typically comes from Dify's own execution logs or webhooks instead.
●OpenClaw is a standalone, always-on agent runtime rather than a library you call, so instrumentation has to happen at the boundary between it and everything else, its API or webhook surface, instead of inline in application code.
In practice, this means "add tracing" means four different things depending on the framework: wrap a callback, wrap a node function, consume an execution log, or hook a webhook. None of that is difficult in isolation. It becomes real, ongoing work when a platform team is the one expected to keep all four producing comparable data.
Why comparable matters more than complete
It's not enough for each framework to be traced somehow. If LangChain traces carry gen_ai.agent.name and LangGraph traces carry graph.node.name with no equivalent, per-agent cost and error-rate comparisons across frameworks silently break. The fix is the same one from the previous post: standardize on a shared attribute schema, such as OpenTelemetry's GenAI semantic conventions, and map every framework's own event model onto it, rather than letting each framework's instrumentation invent its own field names.
What automatic instrumentation is actually buying you
Given that mapping work, "out-of-the-box support for framework X" in an observability tool is worth more than it sounds like on a feature list. It's not just a connector, it's someone else having already done the work of mapping LangChain's callbacks, LangGraph's node/edge model, Dify's execution logs, and OpenClaw's API surface onto one consistent schema, so a platform team isn't building and maintaining four separate adapters by hand.
Limitations
●All four frameworks change quickly. Automatic instrumentation still needs to be re-validated against new framework versions, the same way hand-written instrumentation does.
●A hosted platform like Dify may not expose every internal step to any external collector, in-process or not, so some signals may only ever be as detailed as what the platform's own logs provide.
●Consistent collection solves comparability, not governance. Turning collected data into per-team cost and usage rollups is the labeling and aggregation problem covered in the previous post, not something instrumentation coverage does by itself.
Where Bonree ONE fits
Bonree ONE 4.0's AI Observability capability instruments Python, Node.js, and Java applications, with automatic adaptation for common model-native APIs and out-of-the-box support for LangChain, LangGraph, Dify, and OpenClaw, among other frameworks. See https://en.bonree.com/en/aIObservability and the post https://dev.to/bonree-js/building-ai-observability-for-the-native-stack-architecture-design-and-engineering-practice-from-nc1 for the underlying architecture.
A question for readers
If you're running agents on more than one of these frameworks, how are you keeping their telemetry comparable today: a shared schema you enforce yourselves, a vendor's auto-instrumentation, or have you not tried to reconcile them yet?

observability #opentelemetry #devops #Bonree

Why LLM Agent Token Costs Are Hard to Attribute, and What Telemetry Design Can Do About It

Bonree-Js — Thu, 24 Sep 2026 08:53:49 +0000

AI observability is the practice of collecting traces, metrics, and logs from LLM-backed applications so that latency, quality, and token cost can be explained per model call, per agent, and per conversation, not only per HTTP request.
Disclosure: I work at Bonree, an observability vendor. Most of this post is vendor-neutral; Bonree ONE appears near the end.
The invoice tells you the total, not the cause
One user message to an agent can trigger several model calls, tool calls, and retries. Provider dashboards often aggregate by API key or project, which does not show which agent, which model choice, which team, or which conversation drove the spend. That gap is what telemetry design has to close.
Design principles
●Measure per model call, not per request. A request-level number hides which step in the chain was expensive.
●Split by cardinality. Use metrics for totals, with a small fixed set of labels such as model, agent, team, and token type. Keep unbounded identifiers, such as conversation or user IDs, in traces, where per-conversation detail belongs. An ID used as a metric label creates a new time series for every ID.
●Store token counts, convert to currency later. Prices change. Raw counts stay valid, and a price table applied at analysis time keeps history correct.
●Treat prompt and completion capture as a separate decision. Content often contains customer data and needs its own review.
●Know how your provider counts tokens. Cached and reasoning tokens may be reported differently across providers, so check what each field includes before comparing numbers.
●Make spend attributable to a team, not only a model. Once agent, model, and token-type labels exist on the same metrics, a team-level rollup is a query, not a separate reporting pipeline, and it turns a cost question into a governance one: which teams are running agents, at what volume, and with what effect.
Use a standard path
OpenTelemetry's GenAI semantic conventions define shared names for LLM and agent telemetry. They are still evolving, so pin versions and re-check on upgrade. Keeping collection vendor-neutral also turns the choice of backend into a configuration decision rather than an instrumentation rewrite, which matters to any DevOps or SRE team that expects to change tools someday.
A minimal example
The snippet below is vendor-neutral OpenTelemetry code that illustrates the metrics-versus-traces split. It is not a Bonree ONE configuration. Tested with opentelemetry-sdk 1.44.0 and opentelemetry-semantic-conventions 0.65b0; the GenAI conventions are still evolving, so re-check attribute names when you upgrade.
python

from opentelemetry import trace, metrics

Assumes a TracerProvider and MeterProvider are configured elsewhere.

tracer = trace.get_tracer("agent.instrumentation")
meter = metrics.get_meter("agent.instrumentation")

token_usage = meter.create_histogram(
"gen_ai.client.token.usage", unit="{token}",
description="tokens per call",
)

def call_llm(client, *, model, messages, agent, team, session_id):
with tracer.start_as_current_span(f"chat {model}") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.agent.name", agent)
span.set_attribute("gen_ai.conversation.id", session_id) # span only, never a metric label

    resp = client.chat(model=model, messages=messages)  # stand-in for your SDK call
    span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)

    labels = {"gen_ai.request.model": model, "gen_ai.agent.name": agent, "team": team}
    token_usage.record(resp.usage.input_tokens, {**labels, "gen_ai.token.type": "input"})
    token_usage.record(resp.usage.output_tokens, {**labels, "gen_ai.token.type": "output"})
    return resp

Note that the conversation ID lives on the span only, while team is a small, fixed-cardinality label, so it is safe to put on the metric. In a quick local test with a stub client and two conversations across two teams, the metric produced four series (two teams × input/output), not one per conversation.
Where Bonree ONE fits
Bonree ONE 4.0 includes an AI Observability capability that Bonree describes as providing end-to-end tracing, span-level analysis, token usage monitoring, performance metrics, and session-level context tracking for large-model and agent environments. Bonree ONE also supports standard OpenTelemetry ingestion of traces, metrics, and logs. Whether a team-level rollup like the one above is useful on top of that depends on whether agent and team labels are already being recorded consistently, which is the instrumentation-design question this post is about, not something any specific platform does automatically. See and https://en.bonree.com/en/opentelemetrystandard , or the post https://en.bonree.com/en/blog/aiobservabilityinpracticeinstrumentingagentchainsnotjustapicalls . More at https://en.bonree.com .
A question for readers
How does your team attribute agent token spend today: per API key, per agent, per team, or per conversation?

observability #opentelemetry #devops #Bonree

Building AI Observability for the Native Stack: Architecture Design and Engineering Practice from Bonree ONE 4.0

Bonree-Js — Mon, 14 Sep 2026 07:10:09 +0000

AI is changing observability and operations platforms in two directions at once, and it's easy to conflate them. The first direction is familiar: AI applications — RAG pipelines, LLM-backed customer service agents, coding copilots — are now part of the production stack, and they behave in ways traditional APM was never designed to see. The second direction is less discussed but arguably more consequential: AI is no longer only the thing being monitored. It's increasingly the thing doing the monitoring, and the thing taking action on what it finds.

This article walks through how we approached both problems in Bonree ONE 4.0, the latest release of our observability and AIOps platform. Rather than a feature tour, the goal here is to explain the reasoning behind three capability areas we built — AI Observability, an AI-driven natural-language query layer called SmartAsk, and an autonomous agent workbench called Bonree ONE·Sage AI — and the production challenges that shaped their architecture.

Six problems that don't show up on a status page

A quick example sets the tone for all of them. An insurance customer of ours ran an AI phone agent for customer service. A caller asked how to purchase a policy; the agent misparsed the request and responded with instructions for cancelling one instead. No exception was thrown, no error rate moved, no health check failed — by every conventional signal, the call was a complete success. Hallucination and answer quality are orthogonal to uptime.

That example generalizes into six patterns we kept running into, all of which shaped the architecture below:

AI applications fail silently — confidently wrong output that no uptime or latency metric ever flags.
Agent output gets trusted without being understood — code or decisions an operator can't debug when they eventually break.
Bigger context windows don't remove the need for data governance — we've seen our own tooling get visibly worse at intent recognition once a session accumulates enough unfiltered context, well before hitting a hard token limit.
Autonomy raises the security bar — an agent acts at machine speed, and skills that are individually safe can combine into risks nobody reviewed for.
Cost doesn't fail gracefully — a single agentic query can burn hundreds of thousands of tokens, and unlike a slow response, an overrun is invisible until someone reads the invoice.
Org structures haven't caught up — "AI does everything" scales to a team of one, not a team of hundreds, and how a human and an agent actually share an incident war room is still being worked out in practice.

The rest of this article is about the architecture that came out of treating these as engineering problems rather than caveats: three capability areas in Bonree ONE 4.0 — AI Observability, a natural-language query layer called SmartAsk, and an autonomous agent workbench called Bonree ONE·Sage AI.

AI Observability: treating AI applications as first-class citizens in the trace store

The instinct when a new class of application shows up is to bolt a metric onto an existing dashboard. That doesn't hold up here, because the unit of work itself has changed. A single user request into a RAG or multi-agent application can fan out into a dozen internal steps — retrieval, several sequential or parallel model calls, tool invocations, result synthesis — each with its own latency, token cost, and independent chance of failure. Collapsing all of that into one opaque "request," the way a conventional APM trace would, throws away exactly the information needed to debug it.

Show Image

Collection. Instrumentation covers Python, Node.js, and Java, with automatic adaptation for common model-native APIs and out-of-the-box support for LangChain, LangGraph, Dify, and OpenClaw, among other agent frameworks. It's non-invasive — for a framework that's already supported, there's no code change required to start collecting data, which matters more than it might sound: requiring every team to hand-instrument their LangGraph pipeline before they get any observability is a real adoption barrier, not a minor inconvenience. Collection also speaks OpenTelemetry natively — Traces, Metrics, and Logs travel over OTLP — so AI telemetry isn't a second, proprietary data path sitting next to whatever teams already run for the rest of their stack.

Redaction is handled at collection time, not after storage. Prompts and completions routinely contain customer data, and capturing full input/output at every span means you've effectively built a PII pipeline unless sensitive content is filtered before it lands anywhere persistent. The platform supports configurable masking rules and automatic PII identification and filtering applied at the point of capture.

Processing pipeline. Once collected, data moves through real-time desensitization and format normalization, then metric extraction (performance indicators land in structured storage), then an analysis layer that runs hallucination detection and answer-quality scoring. This last stage is what turns raw traces into something closer to a quality signal rather than just a performance one — necessary precisely because, as the insurance example showed, quality problems don't otherwise register as failures at all.

What you actually see. The platform exposes this data through several coordinated views rather than one dashboard trying to do everything:

An application overview lists every AI service with request volume, error rate, response time, and total token consumption, so an anomalous application is visible at a glance before you drill into anything.
Call chain analysis lists individual AI invocations with full input/output content, response time, and status, filterable by application, trace ID, user ID, or session ID.
Call chain detail is where debugging actually happens: a Call Tree renders the trace as a time-series Gantt chart, with an "ALL" mode showing every span type (HTTP, chain, prompt, llm, parser) and an "LLM" mode that filters down to just the model-related nodes for focused analysis of the reasoning layer. A companion Call Map renders the same trace as a topology graph, with each node showing average response time, request count, and token count. Clicking any node opens a detail panel with four tabs — the input/output content and attributes, a timing breakdown, the code stack that triggered the span, and any errors or logs — which is usually enough to go from "this was slow" to a specific, fixable cause, such as a system prompt that had grown past several thousand tokens after multiple rounds of tool output were appended to context.
A performance view aggregates response time, request count, error count, and model request count with trend comparisons against the prior day.
A token view breaks down total consumption by model, separating input and output tokens, with trend charts over time — the level of granularity needed to answer "which model or which application is actually driving this month's bill," rather than only a top-line total.
A model view aggregates by model rather than by application, so teams running more than one LLM in production can compare call volume, average latency, error rate, and token cost side by side and make an informed choice about which model earns its cost for a given task.
A session view is arguably the most important one for conversational and agentic systems specifically, because the most common quality failure — degradation as context accumulates — is invisible if you only ever look at individual requests. Session analysis aggregates every trace belonging to one multi-turn conversation: total token consumption, trace count, and duration at the list level; a per-trace breakdown showing input preview, response time, and LLM/tool call counts inside a session; a waterfall view of any individual trace's internal span hierarchy, color-coded by node type (Agent, Chain, LLM, Task); and — distinctly useful for multi-agent systems — an Agent collaboration topology that shows, round by round, which node in a multi-agent execution consumed how much time and how many tokens. That last view is what makes it possible to say "the second agent in the chain is where both the latency and the token spend are concentrated" instead of only knowing that a three-hour session was expensive without knowing why.

Alerts integrate directly into the same application detail view, categorized by severity (fatal, critical, warning, general, reminder) with filtering by status and rule type, so a spike surfaced in AI Observability doesn't require switching to a separate alerting tool to investigate.

SmartAsk: making the data conversational without giving up rigor

AI Observability answers "what is happening inside my AI applications." SmartAsk answers a related but different question: "how do I get an answer out of all my observability data — AI-related or not — without writing a query." It's a natural-language interface positioned as an expert-level Q&A layer over the whole platform, not just the AI-observability data described above.

The core technical claim here is "deep data understanding": no data modeling or field mapping step is required before you can ask a question. The system automatically resolves the semantics of metrics, logs, traces, and events, and infers the right data source, filter conditions, and time range from the question itself, rather than requiring the user to specify them explicitly the way a dashboard query builder would. In practice this means the barrier to getting an answer drops from "know the metric name, the right PromQL syntax, and which dashboard has it" to "describe what you want to know."

Rather than starting from a blank prompt every time, the platform ships with more than 30 pre-built scenarios across six categories — health inspection, fault diagnosis, performance optimization, change evaluation, ops governance, and cross-cutting "fusion" scenarios that combine several data types in one answer. These aren't templates in the sense of fixed report formats; they're curated starting points distilled from common patterns across customer deployments, meant to be used as-is or adapted with different targets, time ranges, or thresholds.

Answers come back in a consistent three-part structure: an AI-generated summary in plain language, the underlying raw data, and a visualization — so a reader can trust the headline conclusion, verify it against the actual numbers, or hand the chart to someone else without redoing the analysis. Conversations persist as history that can be continued with follow-up questions, and a query that turns out to be useful more than once can be saved and reused rather than re-typed. Results export to PDF or DOC directly, and — a detail that matters operationally — a one-off query can be promoted into a permanent dashboard widget with one action, so an ad hoc investigation and a recurring monitoring need aren't two different workflows requiring two different tools.

The intended audience is deliberately broad: not just the on-call engineer who already knows the query language, but developers and business stakeholders who need an answer from operational data but have never written PromQL or SQL and shouldn't need to.

Bonree ONE·Sage AI: from answering questions to taking action

SmartAsk gets you an answer. Bonree ONE·Sage AI is built to go further — from a natural-language description of a problem or task to actually carrying it out, using models, tools, knowledge bases, skills, and pre-built or custom agents behind a single conversational interface. The framing we use internally is that this is meant to be the difference between AI as a tool you operate and AI as a colleague that operates alongside you — a distinction that sounds like marketing language until you look at what it requires architecturally, which is considerably more than a chat UI in front of an LLM.

Layered architecture. Bonree ONE·Sage AI is built as seven layers, each addressing a distinct concern:

Show Image

Data source layer — observability signals (logs, metrics, traces, alerts, events), operational assets (CMDB, runbooks/knowledge bases, ITSM tickets), and file assets (scripts, environment variables, credentials). All three categories are treated as inputs an agent might need, not just telemetry.
Connector layer — the Model Context Protocol (MCP) as the standard channel for agent-to-tool and agent-to-system communication, connecting to CMDB and ticketing systems; a CLI layer that does pre-processing before data reaches the model, specifically to reduce how many tokens MCP calls consume — a detail worth noting because it's a direct response to the cost problem described earlier, not a generic engineering nicety; plus conventional API and script access.
Security review layer — every signal and asset entering the system goes through specification validation, permission checks, and a security review before it's usable, and this is deliberately the first line of defense, not a step bolted on after something is already running.
Security sandbox layer — this is the layer that makes autonomous execution tolerable in a production environment. It provides resource, network, filesystem, process, and multi-tenant isolation for every agent execution. Because a complex task involving multiple collaborating agents can legitimately run for minutes or, in some cases, hours, isolation has to hold for the duration of a long-running, potentially unattended job, not just for a single quick call. On top of isolation, there's ingress filtering that blocks malicious instructions before they reach an agent, egress filtering that strips sensitive information from output, execution permission controls with a human-approval gate for consequential actions, and a requirement that agent and skill creation and publishing go through supervisor review before they're available to be invoked at all.
Scheduling and orchestration layer — managed across four dimensions: reliability (fallback strategies, error retry, circuit breaking so one flaky tool call doesn't cascade into a stuck task), validity (schema validation to keep malformed data from silently propagating between steps), agility (a primary agent that interprets intent, decomposes a task, and routes sub-tasks to the right sub-agent, monitoring and correcting course during execution), and coherence (long- and short-term memory management, including memory decay and update mechanisms, so an agent adapts to a user's patterns over time without letting stale context degrade later reasoning — the same accumulation problem described in the AI Observability section, addressed architecturally here rather than just observed).
Interaction control layer — web UI, multi-channel access, alert-triggered execution, scheduled/periodic tasks, and session management, covering both interactive and unattended invocation.
Application scenario layer — the top-level entry points: development and testing, release and change management, inspection and maintenance, emergency recovery, disaster-recovery drills, and ops governance.

Two orchestration modes, chosen per task, not tuned on a dial. For standardized, well-understood work — a routine inspection, a change-approval flow — a fixed workflow is the right tool: explicit steps, predictable execution order, straightforward to audit against, and it fails in enumerable ways. For open-ended fault diagnosis, where the root cause is genuinely unknown at the start and the next useful action depends entirely on what the previous one revealed, a scripted workflow breaks down almost immediately, because you can't pre-write a decision tree for a failure mode you haven't seen yet. That calls for the autonomous-decision mode, where the agent reasons step by step and decides what to check next based on what it just found.

We've watched this second mode play out in a real diagnostic session, and it's a useful illustration of how the security sandbox and orchestration layers work together rather than as separate concerns. An agent was asked to investigate a Java service running in a container and attempt recovery if warranted, with no information given up front about whether the environment was Kubernetes or plain Docker, or where the container lived. It probed the environment, determined it was Docker, and located the target container, then worked through memory, thread, CPU, garbage-collection, and log analysis autonomously — the kind of multi-step investigation that would otherwise require an engineer to log in and do by hand. At two specific points, where the next command would have read deep internal container state, it stopped, displayed the exact command it intended to run, and waited for explicit human confirmation before proceeding. It didn't ask permission for read-only statistics gathering. It asked only for the operations classified as having genuine potential impact — a concrete instance of the "execution permission control + human approval gate" mechanism described in the sandbox layer above, not a one-off safety feature bolted on for that specific session.

Building agents, not just using them. The workbench separates three roles — administrators handle model access and platform-wide security and audit configuration; creators build skills and agents in a dedicated workspace, using either the workflow or autonomous-decision construction method, then publish for their own use or submit for review to share more broadly; users consume whatever has been published, either the built-in library or anything added from an internal resource marketplace. The same person is commonly all three at different times — building a skill in the morning and consuming someone else's in the afternoon — and the platform is built around that overlap rather than assuming rigid role separation.

Out of the box, the platform includes more than 40 MCP-based tools and is compatible with external MCP servers, more than 10 ready-to-use skills including a "deep service diagnostics" skill, and a knowledge base that accepts standard document formats (Markdown, TXT, PDF) as well as API-based ingestion, so an organization's own runbooks and incident playbooks — the kind of institutional knowledge that otherwise lives in a senior engineer's head — can be folded directly into what an agent draws on. Several pre-built expert agents ship with the platform, including a terminal/host diagnostics agent and a database-specialist agent built on patterns distilled from experienced DBA troubleshooting workflows, usable directly or scheduled to run as an unattended, always-on specialist.

Why this is worth the architectural complexity. The value case breaks into four tiers that build on each other. The most direct is efficiency: routine inspection, troubleshooting, reporting, and alert handling can run end-to-end through an agent, freeing operators from repetitive work and materially shortening mean time to resolution once a team can pull full-stack data with one query instead of stitching it together by hand. The second is an asset value that's easy to undervalue until you've lost it: a senior engineer's troubleshooting instinct, previously undocumented and lost when they leave or change roles, gets encoded as a reusable skill or knowledge-base entry instead — turning tacit, personal know-how into a structured, transferable asset. The third is a shift in operating model, from reactive to anticipatory: scheduled inspection combined with baseline analysis surfaces risk before it becomes an incident, and direct integration with ticketing and CMDB systems closes the loop from detection through resolution rather than stopping at notification. The fourth is more strategic than operational: full execution logging supports the audit requirements that regulated industries — finance, government — actually need, which is a precondition for deploying autonomous agents in those environments at all, not an optional nice-to-have.

What's still unresolved

It would be dishonest to present all of this as a solved problem, and a few things are worth naming directly.

Skill-level security review is necessary but not sufficient. A skill that queries a monitoring API and a skill that restarts a process can each look completely safe in isolation and still combine into something neither reviewer anticipated once an agent chains them together at runtime in a sequence nobody explicitly tested. Static, pre-publish review catches the safety of individual components; it does not catch emergent risk from composition. The mitigations available so far — runtime ingress/egress filtering that doesn't depend on having anticipated the specific dangerous combination in advance, and treating the human-confirmation gate as a backstop rather than a complete answer — are partial, not complete, and we don't think this is a solved problem industry-wide.

Cost governance is a genuine, ongoing tension rather than a one-time optimization. The internal example cited earlier — token spend growing substantially even as the explicit goal of adopting AI tooling was cost reduction — isn't unusual, and treating token consumption as an engineering-visible signal (per model, per session, per conversation turn, as described in the AI Observability section) is necessary precisely because that tension doesn't resolve itself; it has to be actively monitored and traded off against agent autonomy on purpose.

And organizationally, the question of how a human on-call engineer and an autonomous diagnostic agent actually collaborate during a live incident — who has authority to decide, how escalation works when the agent's confidence is low, what a shared "war room" looks like when one participant is a model — is still being worked out in practice at most organizations we've talked to, ourselves included. Tooling can support that collaboration, but it can't yet fully define it.

Where this leaves the three pieces

AI Observability, SmartAsk, and Bonree ONE·Sage AI aren't three independent features so much as three layers of the same problem. Observability tells you what's actually happening — including inside the AI applications that used to be a blind spot entirely. SmartAsk makes that information reachable in natural language, without requiring every stakeholder to learn a query syntax first. And Bonree ONE·Sage AI is where understanding turns into action, under a governance model — layered isolation, staged review, and a confirmation gate scoped specifically to consequential operations — built for the fact that the system doing the acting is no longer only a human being.

None of this makes AI-native operations a solved problem. But it's a different, and more specific, problem than "add a chatbot to the dashboard," and the architecture ends up looking correspondingly different once you take that seriously.