AI is changing observability and operations platforms in two directions at once, and it's easy to conflate them. The first direction is familiar: AI applications — RAG pipelines, LLM-backed customer service agents, coding copilots — are now part of the production stack, and they behave in ways traditional APM was never designed to see. The second direction is less discussed but arguably more consequential: AI is no longer only the thing being monitored. It's increasingly the thing doing the monitoring, and the thing taking action on what it finds.
This article walks through how we approached both problems in Bonree ONE 4.0, the latest release of our observability and AIOps platform. Rather than a feature tour, the goal here is to explain the reasoning behind three capability areas we built — AI Observability, an AI-driven natural-language query layer called SmartAsk, and an autonomous agent workbench called Bonree ONE·Sage AI — and the production challenges that shaped their architecture.
Six problems that don't show up on a status page
A quick example sets the tone for all of them. An insurance customer of ours ran an AI phone agent for customer service. A caller asked how to purchase a policy; the agent misparsed the request and responded with instructions for cancelling one instead. No exception was thrown, no error rate moved, no health check failed — by every conventional signal, the call was a complete success. Hallucination and answer quality are orthogonal to uptime.
That example generalizes into six patterns we kept running into, all of which shaped the architecture below:
AI applications fail silently — confidently wrong output that no uptime or latency metric ever flags.
Agent output gets trusted without being understood — code or decisions an operator can't debug when they eventually break.
Bigger context windows don't remove the need for data governance — we've seen our own tooling get visibly worse at intent recognition once a session accumulates enough unfiltered context, well before hitting a hard token limit.
Autonomy raises the security bar — an agent acts at machine speed, and skills that are individually safe can combine into risks nobody reviewed for.
Cost doesn't fail gracefully — a single agentic query can burn hundreds of thousands of tokens, and unlike a slow response, an overrun is invisible until someone reads the invoice.
Org structures haven't caught up — "AI does everything" scales to a team of one, not a team of hundreds, and how a human and an agent actually share an incident war room is still being worked out in practice.
The rest of this article is about the architecture that came out of treating these as engineering problems rather than caveats: three capability areas in Bonree ONE 4.0 — AI Observability, a natural-language query layer called SmartAsk, and an autonomous agent workbench called Bonree ONE·Sage AI.
AI Observability: treating AI applications as first-class citizens in the trace store
The instinct when a new class of application shows up is to bolt a metric onto an existing dashboard. That doesn't hold up here, because the unit of work itself has changed. A single user request into a RAG or multi-agent application can fan out into a dozen internal steps — retrieval, several sequential or parallel model calls, tool invocations, result synthesis — each with its own latency, token cost, and independent chance of failure. Collapsing all of that into one opaque "request," the way a conventional APM trace would, throws away exactly the information needed to debug it.
Show Image
Collection. Instrumentation covers Python, Node.js, and Java, with automatic adaptation for common model-native APIs and out-of-the-box support for LangChain, LangGraph, Dify, and OpenClaw, among other agent frameworks. It's non-invasive — for a framework that's already supported, there's no code change required to start collecting data, which matters more than it might sound: requiring every team to hand-instrument their LangGraph pipeline before they get any observability is a real adoption barrier, not a minor inconvenience. Collection also speaks OpenTelemetry natively — Traces, Metrics, and Logs travel over OTLP — so AI telemetry isn't a second, proprietary data path sitting next to whatever teams already run for the rest of their stack.
Redaction is handled at collection time, not after storage. Prompts and completions routinely contain customer data, and capturing full input/output at every span means you've effectively built a PII pipeline unless sensitive content is filtered before it lands anywhere persistent. The platform supports configurable masking rules and automatic PII identification and filtering applied at the point of capture.
Processing pipeline. Once collected, data moves through real-time desensitization and format normalization, then metric extraction (performance indicators land in structured storage), then an analysis layer that runs hallucination detection and answer-quality scoring. This last stage is what turns raw traces into something closer to a quality signal rather than just a performance one — necessary precisely because, as the insurance example showed, quality problems don't otherwise register as failures at all.
What you actually see. The platform exposes this data through several coordinated views rather than one dashboard trying to do everything:
An application overview lists every AI service with request volume, error rate, response time, and total token consumption, so an anomalous application is visible at a glance before you drill into anything.
Call chain analysis lists individual AI invocations with full input/output content, response time, and status, filterable by application, trace ID, user ID, or session ID.
Call chain detail is where debugging actually happens: a Call Tree renders the trace as a time-series Gantt chart, with an "ALL" mode showing every span type (HTTP, chain, prompt, llm, parser) and an "LLM" mode that filters down to just the model-related nodes for focused analysis of the reasoning layer. A companion Call Map renders the same trace as a topology graph, with each node showing average response time, request count, and token count. Clicking any node opens a detail panel with four tabs — the input/output content and attributes, a timing breakdown, the code stack that triggered the span, and any errors or logs — which is usually enough to go from "this was slow" to a specific, fixable cause, such as a system prompt that had grown past several thousand tokens after multiple rounds of tool output were appended to context.
A performance view aggregates response time, request count, error count, and model request count with trend comparisons against the prior day.
A token view breaks down total consumption by model, separating input and output tokens, with trend charts over time — the level of granularity needed to answer "which model or which application is actually driving this month's bill," rather than only a top-line total.
A model view aggregates by model rather than by application, so teams running more than one LLM in production can compare call volume, average latency, error rate, and token cost side by side and make an informed choice about which model earns its cost for a given task.
A session view is arguably the most important one for conversational and agentic systems specifically, because the most common quality failure — degradation as context accumulates — is invisible if you only ever look at individual requests. Session analysis aggregates every trace belonging to one multi-turn conversation: total token consumption, trace count, and duration at the list level; a per-trace breakdown showing input preview, response time, and LLM/tool call counts inside a session; a waterfall view of any individual trace's internal span hierarchy, color-coded by node type (Agent, Chain, LLM, Task); and — distinctly useful for multi-agent systems — an Agent collaboration topology that shows, round by round, which node in a multi-agent execution consumed how much time and how many tokens. That last view is what makes it possible to say "the second agent in the chain is where both the latency and the token spend are concentrated" instead of only knowing that a three-hour session was expensive without knowing why.
Alerts integrate directly into the same application detail view, categorized by severity (fatal, critical, warning, general, reminder) with filtering by status and rule type, so a spike surfaced in AI Observability doesn't require switching to a separate alerting tool to investigate.
SmartAsk: making the data conversational without giving up rigor
AI Observability answers "what is happening inside my AI applications." SmartAsk answers a related but different question: "how do I get an answer out of all my observability data — AI-related or not — without writing a query." It's a natural-language interface positioned as an expert-level Q&A layer over the whole platform, not just the AI-observability data described above.
The core technical claim here is "deep data understanding": no data modeling or field mapping step is required before you can ask a question. The system automatically resolves the semantics of metrics, logs, traces, and events, and infers the right data source, filter conditions, and time range from the question itself, rather than requiring the user to specify them explicitly the way a dashboard query builder would. In practice this means the barrier to getting an answer drops from "know the metric name, the right PromQL syntax, and which dashboard has it" to "describe what you want to know."
Rather than starting from a blank prompt every time, the platform ships with more than 30 pre-built scenarios across six categories — health inspection, fault diagnosis, performance optimization, change evaluation, ops governance, and cross-cutting "fusion" scenarios that combine several data types in one answer. These aren't templates in the sense of fixed report formats; they're curated starting points distilled from common patterns across customer deployments, meant to be used as-is or adapted with different targets, time ranges, or thresholds.
Answers come back in a consistent three-part structure: an AI-generated summary in plain language, the underlying raw data, and a visualization — so a reader can trust the headline conclusion, verify it against the actual numbers, or hand the chart to someone else without redoing the analysis. Conversations persist as history that can be continued with follow-up questions, and a query that turns out to be useful more than once can be saved and reused rather than re-typed. Results export to PDF or DOC directly, and — a detail that matters operationally — a one-off query can be promoted into a permanent dashboard widget with one action, so an ad hoc investigation and a recurring monitoring need aren't two different workflows requiring two different tools.
The intended audience is deliberately broad: not just the on-call engineer who already knows the query language, but developers and business stakeholders who need an answer from operational data but have never written PromQL or SQL and shouldn't need to.
Bonree ONE·Sage AI: from answering questions to taking action
SmartAsk gets you an answer. Bonree ONE·Sage AI is built to go further — from a natural-language description of a problem or task to actually carrying it out, using models, tools, knowledge bases, skills, and pre-built or custom agents behind a single conversational interface. The framing we use internally is that this is meant to be the difference between AI as a tool you operate and AI as a colleague that operates alongside you — a distinction that sounds like marketing language until you look at what it requires architecturally, which is considerably more than a chat UI in front of an LLM.
Layered architecture. Bonree ONE·Sage AI is built as seven layers, each addressing a distinct concern:
Show Image
Data source layer — observability signals (logs, metrics, traces, alerts, events), operational assets (CMDB, runbooks/knowledge bases, ITSM tickets), and file assets (scripts, environment variables, credentials). All three categories are treated as inputs an agent might need, not just telemetry.
Connector layer — the Model Context Protocol (MCP) as the standard channel for agent-to-tool and agent-to-system communication, connecting to CMDB and ticketing systems; a CLI layer that does pre-processing before data reaches the model, specifically to reduce how many tokens MCP calls consume — a detail worth noting because it's a direct response to the cost problem described earlier, not a generic engineering nicety; plus conventional API and script access.
Security review layer — every signal and asset entering the system goes through specification validation, permission checks, and a security review before it's usable, and this is deliberately the first line of defense, not a step bolted on after something is already running.
Security sandbox layer — this is the layer that makes autonomous execution tolerable in a production environment. It provides resource, network, filesystem, process, and multi-tenant isolation for every agent execution. Because a complex task involving multiple collaborating agents can legitimately run for minutes or, in some cases, hours, isolation has to hold for the duration of a long-running, potentially unattended job, not just for a single quick call. On top of isolation, there's ingress filtering that blocks malicious instructions before they reach an agent, egress filtering that strips sensitive information from output, execution permission controls with a human-approval gate for consequential actions, and a requirement that agent and skill creation and publishing go through supervisor review before they're available to be invoked at all.
Scheduling and orchestration layer — managed across four dimensions: reliability (fallback strategies, error retry, circuit breaking so one flaky tool call doesn't cascade into a stuck task), validity (schema validation to keep malformed data from silently propagating between steps), agility (a primary agent that interprets intent, decomposes a task, and routes sub-tasks to the right sub-agent, monitoring and correcting course during execution), and coherence (long- and short-term memory management, including memory decay and update mechanisms, so an agent adapts to a user's patterns over time without letting stale context degrade later reasoning — the same accumulation problem described in the AI Observability section, addressed architecturally here rather than just observed).
Interaction control layer — web UI, multi-channel access, alert-triggered execution, scheduled/periodic tasks, and session management, covering both interactive and unattended invocation.
Application scenario layer — the top-level entry points: development and testing, release and change management, inspection and maintenance, emergency recovery, disaster-recovery drills, and ops governance.
Two orchestration modes, chosen per task, not tuned on a dial. For standardized, well-understood work — a routine inspection, a change-approval flow — a fixed workflow is the right tool: explicit steps, predictable execution order, straightforward to audit against, and it fails in enumerable ways. For open-ended fault diagnosis, where the root cause is genuinely unknown at the start and the next useful action depends entirely on what the previous one revealed, a scripted workflow breaks down almost immediately, because you can't pre-write a decision tree for a failure mode you haven't seen yet. That calls for the autonomous-decision mode, where the agent reasons step by step and decides what to check next based on what it just found.
We've watched this second mode play out in a real diagnostic session, and it's a useful illustration of how the security sandbox and orchestration layers work together rather than as separate concerns. An agent was asked to investigate a Java service running in a container and attempt recovery if warranted, with no information given up front about whether the environment was Kubernetes or plain Docker, or where the container lived. It probed the environment, determined it was Docker, and located the target container, then worked through memory, thread, CPU, garbage-collection, and log analysis autonomously — the kind of multi-step investigation that would otherwise require an engineer to log in and do by hand. At two specific points, where the next command would have read deep internal container state, it stopped, displayed the exact command it intended to run, and waited for explicit human confirmation before proceeding. It didn't ask permission for read-only statistics gathering. It asked only for the operations classified as having genuine potential impact — a concrete instance of the "execution permission control + human approval gate" mechanism described in the sandbox layer above, not a one-off safety feature bolted on for that specific session.
Building agents, not just using them. The workbench separates three roles — administrators handle model access and platform-wide security and audit configuration; creators build skills and agents in a dedicated workspace, using either the workflow or autonomous-decision construction method, then publish for their own use or submit for review to share more broadly; users consume whatever has been published, either the built-in library or anything added from an internal resource marketplace. The same person is commonly all three at different times — building a skill in the morning and consuming someone else's in the afternoon — and the platform is built around that overlap rather than assuming rigid role separation.
Out of the box, the platform includes more than 40 MCP-based tools and is compatible with external MCP servers, more than 10 ready-to-use skills including a "deep service diagnostics" skill, and a knowledge base that accepts standard document formats (Markdown, TXT, PDF) as well as API-based ingestion, so an organization's own runbooks and incident playbooks — the kind of institutional knowledge that otherwise lives in a senior engineer's head — can be folded directly into what an agent draws on. Several pre-built expert agents ship with the platform, including a terminal/host diagnostics agent and a database-specialist agent built on patterns distilled from experienced DBA troubleshooting workflows, usable directly or scheduled to run as an unattended, always-on specialist.
Why this is worth the architectural complexity. The value case breaks into four tiers that build on each other. The most direct is efficiency: routine inspection, troubleshooting, reporting, and alert handling can run end-to-end through an agent, freeing operators from repetitive work and materially shortening mean time to resolution once a team can pull full-stack data with one query instead of stitching it together by hand. The second is an asset value that's easy to undervalue until you've lost it: a senior engineer's troubleshooting instinct, previously undocumented and lost when they leave or change roles, gets encoded as a reusable skill or knowledge-base entry instead — turning tacit, personal know-how into a structured, transferable asset. The third is a shift in operating model, from reactive to anticipatory: scheduled inspection combined with baseline analysis surfaces risk before it becomes an incident, and direct integration with ticketing and CMDB systems closes the loop from detection through resolution rather than stopping at notification. The fourth is more strategic than operational: full execution logging supports the audit requirements that regulated industries — finance, government — actually need, which is a precondition for deploying autonomous agents in those environments at all, not an optional nice-to-have.
What's still unresolved
It would be dishonest to present all of this as a solved problem, and a few things are worth naming directly.
Skill-level security review is necessary but not sufficient. A skill that queries a monitoring API and a skill that restarts a process can each look completely safe in isolation and still combine into something neither reviewer anticipated once an agent chains them together at runtime in a sequence nobody explicitly tested. Static, pre-publish review catches the safety of individual components; it does not catch emergent risk from composition. The mitigations available so far — runtime ingress/egress filtering that doesn't depend on having anticipated the specific dangerous combination in advance, and treating the human-confirmation gate as a backstop rather than a complete answer — are partial, not complete, and we don't think this is a solved problem industry-wide.
Cost governance is a genuine, ongoing tension rather than a one-time optimization. The internal example cited earlier — token spend growing substantially even as the explicit goal of adopting AI tooling was cost reduction — isn't unusual, and treating token consumption as an engineering-visible signal (per model, per session, per conversation turn, as described in the AI Observability section) is necessary precisely because that tension doesn't resolve itself; it has to be actively monitored and traded off against agent autonomy on purpose.
And organizationally, the question of how a human on-call engineer and an autonomous diagnostic agent actually collaborate during a live incident — who has authority to decide, how escalation works when the agent's confidence is low, what a shared "war room" looks like when one participant is a model — is still being worked out in practice at most organizations we've talked to, ourselves included. Tooling can support that collaboration, but it can't yet fully define it.
Where this leaves the three pieces
AI Observability, SmartAsk, and Bonree ONE·Sage AI aren't three independent features so much as three layers of the same problem. Observability tells you what's actually happening — including inside the AI applications that used to be a blind spot entirely. SmartAsk makes that information reachable in natural language, without requiring every stakeholder to learn a query syntax first. And Bonree ONE·Sage AI is where understanding turns into action, under a governance model — layered isolation, staged review, and a confirmation gate scoped specifically to consequential operations — built for the fact that the system doing the acting is no longer only a human being.
None of this makes AI-native operations a solved problem. But it's a different, and more specific, problem than "add a chatbot to the dashboard," and the architecture ends up looking correspondingly different once you take that seriously.
Top comments (0)