DEV Community: ObservabilityGuy

Alibaba Cloud Launches Full-stack AgenticOps, Making the Agent the First User in Operations and R&D

ObservabilityGuy — Thu, 12 Nov 2026 05:16:46 +0000

This article introduces Alibaba Cloud's full-stack AgenticOps platform, enabling autonomous, closed-loop AI operations and R&D through unified observability, evaluation, and controlled execution.

I. Three Walls and a Changing First User

As large language models (LLMs) and Agents move from demos into production, more and more enterprises want the same thing: to run Agents stably in production, not just prove them in a demo. Once an Agent touches real business, two problems surface at once. First, metrics, logs, traces, and events are scattered across systems, and getting from an anomaly to its root cause still depends on a handful of experts. Second, the Agent's own behavior is hard to explain, its effectiveness hard to evaluate, bad cases hard to retain, and successful practices hard to replicate. Together these become the three walls AgenticOps has to cross: data that cannot be collected, context that cannot be understood, and capabilities that cannot be verified.

The root of the change is a changing first user. Operations is moving from manual operations through artificial intelligence for IT operations (AIOps) and Q&A assistants toward goal-driven, autonomous, closed-loop AgenticOps. According to IDC's “Agent Adoption” report, the number of actively deployed AI Agents worldwide is growing roughly 40-fold over four years. The primary consumer of the data is no longer a person, and the responsibility of observability is extending in two directions: supplying trusted facts to Agents, and bringing the Agent's own operational data into observability.

To meet these challenges, Alibaba Cloud is releasing full-stack AgenticOps capabilities. It first extends the observability boundary so that the system is legible to Agents and the Agent itself is legible as well, laying a trusted, understandable foundation of facts. It then lets Agents perceive, judge, act under control, and keep evolving on that foundation, turning “seen” into “solved” as one continuous chain. The capabilities themselves are also being open-sourced for the industry to build and verify together. Concretely, the goal is to move operations and R&D from human-led, tool-assisted work toward goal-driven, Agent-led autonomy.

II. Expanding the Observability Boundary: Making the System and the Agent Legible

When the Agent becomes the first user, the observability boundary has to expand in two directions at once. Downward, production facts from cross-cloud, cross-stack, and multi-cloud environments must be organized into trusted data an Agent can consume directly. Upward, the Agent's own traces, cost, quality, and risk must come under observability as well. Cloud Monitor 2.0 and AgentLoop carry this layer: the former is a unified, full-domain observability platform built for AgenticOps, and the latter turns the Agent into an object that can be observed, evaluated, and evolved.

Cloud Monitor 2.0 Agentic Observability: A Unified Full-domain Observability Platform

The upgraded Cloud Monitor 2.0, now with agentic observability, makes both the system and Agents legible. A complete Agent task breaks into three consecutive phases: trusted facts, evidence-based judgment, and controlled action. For observability by Agents, Cloud Monitor 2.0 hands production facts to the Agent through a single pipeline.

For integration, intelligent onboarding supports more than 500 cloud services and components and works across clouds and accounts. It extends the observability scope from Agents, endpoints, and applications to middleware, infrastructure, multi-cloud environments, and open-source data sources. A five-step loop — intent input, intelligent discovery, solution generation, Preview authorization, and deployment with validation — turns onboarding from form-filling into a goal-driven task, and the finish line is real, usable data rather than a successful configuration.

For governance, the observability pipeline runs through three stages — standardization, organization, and trustworthiness — turning raw observability data such as metrics, logs, traces, events, and metadata into production facts that carry their source, freshness, semantics, quality, and cost boundaries.

For semantics, UModel builds a complete semantic map of production systems. EntitySet describes entities; LogSet, MetricSet, TraceSet, and other sets organize data; relationship links connect dependencies; RunbookSet retains diagnostic and operational knowledge; and SPL provides one unified query language. Its more than 3,200 built-in model definitions cover 10 observability domains and 214 model groups, so an Agent knows exactly which object each piece of data points to. On top of that, Intelligent Inspection covers 117 types of production entities, and built-in expert Skills can be switched on with one click and keep investigating. Root cause analysis (RCA) becomes a verifiable closed loop: investigations driven by a system map, hypotheses and counter-evidence unfolded dynamically, evidence-based evaluation, and online bad cases fed back into regression. When evidence is insufficient, the system keeps investigating rather than forcing a verdict beyond the evidence.

Entry points and security decide whether an Agent can truly enter production. Cloud Monitor 2.0 opens its capabilities through three entry points — Skill at the method layer, the CMS2 command-line interface (CLI) at the execution layer, and the Model Context Protocol (MCP) at the connection layer via the CMS Cloud MCP connector — all sharing one foundation that covers the full data lifecycle of ingestion, computation, storage, and consumption. All three follow the same security contract: queries and investigations are read-only by default and are opened up first; writes and automated execution require separate authorization, with a Preview of the change scope, explicit confirmation, and a read-back check after execution; and Agent permissions never exceed those of the initiating user, with identity, scope, parameters, and results logged throughout. Leave reasoning and judgment to the model; leave the facts to Cloud Monitor 2.0.

For observability of Agents, Cloud Monitor 2.0 also brings the Agent itself into view. To address three hard problems — stitching full-stack traces together, explaining execution, and drilling down to the root cause of anomalies — it provides full-stack AI observability across the user layer, Agent Runtime, model layer, platform layer, and AI infrastructure, with AI Agent observability, inference service observability, model insights, Sandbox observability, and GPU observability covering each layer. The data produced by observing Agents is the same set of facts that Agents rely on to observe systems; the two capabilities share one foundation.

AgentLoop: Turning Agent Runtime Traces into Observability Assets That Can Be Evaluated and Evolved

Cloud Monitor 2.0 helps an Agent understand the system; AgentLoop turns the lens on the Agent itself, so every run is visible, accurately evaluated, and open to improvement. Alibaba Cloud's view is that Agent effectiveness is not something you tune; it is something you run, evaluate, improve, and regress. The divide between a demo and a production-grade Agent is whether, when real-world tool failures, cost swings, and quality regressions show up, there is a platform mechanism for observability, evaluation, optimization, and regression that turns Agent observability into a continuously evolving data asset. As an enterprise Agent observability and optimization platform, AgentLoop serves production Agent environments along four lines: full-stack observability, effectiveness evaluation, continuous optimization, and experiment regression.

On the collection side, LoongSuite x AgentSight combines instrumentation agents (probes) for four languages with eBPF zero-code instrumentation, covering more than 40 frameworks without duplication and onboarding everything from LangChain, AgentScope, and Dify to Claude Code, Cursor, and MCP. It turns the Agent's reasoning traces, token cost, and tool calls into analyzable data. Behavioral auditing then upgrades alert lists into explainable, reviewable, actionable risk events, so operations can pick real risk out of the noise.

Above observability, AgentLoop turns data into judgment and improvement. Agent-as-a-Judge uses an Agent as the evaluator: it loads tools and Skills, learns enterprise private knowledge, and returns its reasons along with a traceable record of each run, pushing automated evaluation into a confidence range close to expert judgment. With a second review by human experts, evaluation accuracy reaches over 90%. Attribution analysis turns scores into concrete improvements, locating them across prompt, tool, knowledge, planning, memory, model, and policy. Experiment regression runs five steps — locked baseline, offline regression, small-traffic A/B testing, change gate, and canary release — so every change is a positive optimization; if any core metric misses its threshold, the change is blocked or rolled back.

Once Agents can evolve themselves, the moat in enterprise AI shifts from whose model is better to whose evolution flywheel spins faster.

III. Closed-loop AIOps: STAROps Turns Facts into Controlled Action

Once the observability boundary has expanded and the facts are clear, the operations domain has to turn judgment into controlled action. As Alibaba Cloud's unified entry point for AIOps, STAROps unifies operational context and embeds cross-domain analysis, diagnostics, and continuous closed-loop remediation into an enterprise's existing workspaces and Agent ecosystem. Underneath, the STAROps Agentic Ops Engine drives a single chain: intent understanding, dynamic data perception, full-domain context assembly, task decomposition, intent routing, unified conclusion analysis, execution and validation, and self-evolution.

That chain rests on a unified operational context. Unified Connectors reach across Alibaba Cloud accounts, enterprise IDCs, and external systems; unified data modeling then brings metrics, logs, traces, events, changes, and code together with applications, instances, containers, and databases into one context that an Agent can understand and investigate.

In practice, a single question is enough to know and understand operational data. According to the team, a user can simply ask, “Show me the applications with relatively high error rates in the last hour” — an application performance monitoring (APM) question — and the system queries across metrics, logs, traces, resources, and changes, then returns a ranked list and key findings: for example, that the application named “review” has an error rate of 0.367%, about 30 times that of the runner-up, frontend-proxy. For end-to-end diagnostics, STAROps investigates in a hypothesis-driven way, converging through scope locking, full-domain evidence correlation, hypothesis validation, and conclusion output, and covers cross-cloud products, change anomalies, end-to-end application issues, and more.

Continuous loops and controlled execution are key to STAROps. Long-running tasks keep working toward operational goals, find problems proactively, and turn resolution traces and validation results into reusable experience. Remediation follows a tiered strategy: low-risk queries pass automatically; critical changes such as configuration, restarts, and scaling need human confirmation; operations that match a prohibition rule are blocked outright; and every action is checked twice, against execution policy and Resource Access Management (RAM) permissions, so the degree of automation always matches production risk and the whole process stays traceable.

IV. Faster Intelligent Development: Alibaba Cloud DevOps R&D Agent Lets a Digital Avatar Work Autonomously

Alibaba Cloud is also extending Agent capabilities from operations into R&D delivery. The Alibaba Cloud DevOps R&D Agent, released at this event, is built on the STAROps digital employee and has three clear goals: intelligent coverage of every scenario, autonomous Agent work, and automatic maintenance of team knowledge. For the user it is a dedicated digital avatar with the same Alibaba Cloud DevOps resource permissions as the person it represents, so there is no separate permission model to maintain; it can commit code, update requirements, and handle other day-to-day R&D work on the user's behalf.

R&D coverage spans the full chain, from requirement to release. The Agent covers requirement breakdown and test case generation, intelligent coding, continuous integration (CI) failure analysis and auto-repair, merge request (MR) review, functional acceptance, and release risk evaluation — every step from requirement and coding to testing, acceptance, and release. At each step it works autonomously within team standards, and a developer only has to review and confirm. Traditional DevOps runs these steps serially and depends on people to push them forward; with an Agent in the loop, multiple workstreams run in parallel and delivery speeds up noticeably.
The Agent works on its own, from scheduled inspections to event triggers. It responds to user instructions, and it can also be woken on a schedule — to generate a daily iteration progress report or assess pipeline health weekly — or by an event, stepping in to analyze a newly assigned work item or MR, or a failed pipeline run. R&D collaboration shifts from people looking for work to work finding people.
Team knowledge accumulates by itself, and the Agent gets smarter with use. Every interaction becomes team knowledge, stored in code repositories and recalled when later tasks need it. Scripts recorded during acceptance testing, for example, are filed automatically and reused the next time the build is submitted for testing, which means faster runs, less manual work, and lower token consumption. The more it is used, the more knowledge it accumulates and the more efficient collaboration becomes: this is a self-evolving R&D Agent.

V. Open-source Collaboration: Making the AgenticOps Foundation Reusable, Verifiable, and Co-built

Cloud Monitor 2.0, AgentLoop, STAROps, and Alibaba Cloud DevOps are product capabilities for enterprise production. Beyond them, Alibaba Cloud contributes to three open-source projects — OpenTelemetry (OTel), UModel, and BRiSE — to break, at the foundation, the three walls of data that cannot be collected, context that cannot be understood, and capabilities that cannot be verified, so the whole stack becomes a public asset the industry can reuse, verify, and build together.

On collection, OTel provides the standard and LoongSuite fills the scenario gaps and gives back, which is how the collection wall comes down. For AI applications whose code cannot be changed, OpenTelemetry eBPF Instrumentation (OBI) collects at the kernel with eBPF and restores plaintext inside the TLS boundary, turning Agents written in any language into standard OTel GenAI semantics with zero code changes, zero SDKs, and zero restarts. For applications built on high-code frameworks, instrumentation agents for Java, Go, Python, and other languages instrument at compile time and at runtime to fill in object semantics. Apache Dubbo is one example: the team started from production issues, fixed incorrect traces, completed metrics and field specifications, and reached stable support, then extended the same path to more domestic frameworks such as MyBatis and XXL-JOB. Over three years this work has produced 432 contributions, 205 of them merged PRs at a 79.2% acceptance rate, and the team has earned five governance seats across the Go, Java, and GenAI semantic convention projects, so production experience keeps feeding into OTel rules.
On semantics, UModel, the unified semantic model in Cloud Monitor 2.0, is opening further to the industry as an open-source project, and it addresses the problem of context that cannot be understood. It centers on fault objects and aligns entities, relationships, and time ranges across metrics, logs, requests, and events. In May 2026, UModel was open-sourced under the Apache-2.0 license, and Alibaba Cloud launched an industry initiative to define a universal semantic standard for enterprises (USS), together with XPeng, Zhuoyu Technology, Chanjet, JLC Technology, and Shenzhou Shanglong.
On evaluation, BRiSE fixes the fault environment, cases, and scoring criteria so AgenticOps capabilities can be reproduced, compared, and put into version regression under identical conditions — aimed squarely at the third wall, capabilities that cannot be verified. The benchmark is layered by evidence complexity: data querying has accumulated 1,360 questions, and inspection and anomaly analysis have built 418 tasks across 11 evidence sources, 19 technical domains, and 1,576 observable evidence signals; real fault diagnosis has accumulated 764 fault cases covering 145 fault types and more than 20 cloud products. The first batches, RCA100 and RCA200, are now open source, and the benchmark is being co-built with industry and research partners — China Academy of Information and Communications Technology (CAICT), Institute of Software, Chinese Academy of Sciences (ISCAS), Computer Network Information Center, Chinese Academy of Sciences (CNIC), Tsinghua University, Fudan University, and Nankai University — so it keeps moving closer to real scenarios and real customer business.

VI. Conclusion: From Visibility to Decisions and Self-evolution

As LLMs and Agents accelerate into use, Alibaba Cloud offers enterprises a complete infrastructure that spans observability and evaluation through to decision and action. Cloud Monitor 2.0 and AgentLoop first expand the observability boundary and strengthen the foundation, so the system is legible and the Agent is legible. STAROps closes the AIOps loop on that foundation, moving operations from seeing to understanding and controlled decision-making. The Alibaba Cloud DevOps R&D Agent brings the same autonomous loop into daily R&D. And the open-source foundations of OTel, UModel, and BRiSE hand data collection, semantic unification, and capability evaluation to the industry to build together.

When a system can perceive, decide, and turn every resolution into experience for the next one, the competitive question in enterprise AI becomes simple: who evolves faster.

Stuck on 'Connecting'? Troubleshoot AI Voice WebSocket Connection Issues with RUM

ObservabilityGuy — Tue, 10 Nov 2026 07:01:26 +0000

This article introduces troubleshooting WebSocket connection issues in AI voice applications using Alibaba Cloud RUM custom resources and client-to-cloud trace correlation.

1. Connection Establishment Problems in AI Voice Interaction

Picture a voice assistant running on smart glasses: the glasses capture and play audio, and a companion phone app connects to a cloud voice service over WebSocket. The user presses the voice button on the glasses and asks "Do I need an umbrella today?" — and still no reply arrives. Opening the app shows a page stuck on "Connecting", and a retry sometimes brings it back. Model-side metrics look normal, and no failed request in the standard APIs matches that action. Where do you start?

In an application that uses WebSocket, that wait may happen during connection establishment, before any audio goes out. This article records the handshake as a custom resource in Cloud Monitor 2.0 Real User Monitoring (RUM) to confirm the connection result, the duration of the attempt, and which user environments are affected. Once client-to-cloud tracing is in place, you can use the trace to inspect the corresponding server-side processing.

A voice assistant has to stream audio to the cloud continuously, receive results, and handle user interruptions, so WebSocket fits this kind of bidirectional interaction.

The example here uses a cascaded voice architecture built from automatic speech recognition (ASR), an Agent, a model, and text-to-speech (TTS); a real-time multimodal model may merge some of these stages.

2. Handshake Records: Connection Result and User Environment

Continuing with the scenario above: first confirm that this voice action opened a new connection instead of reusing an existing one, then find the matching handshake record in the user sessions and actions already collected. Suppose the record shows the following result:

Field	Example value	What it tells you
`success`	`false`	This handshake did not succeed
`statusCode`	`403`	The server returned an HTTP rejection
`errorMessage`	`auth-rejected`	Classified per this article's convention; the exact reason for rejection still needs checking

This record narrows a vague "the AI didn't respond" down to a rejected handshake, which moves troubleshooting to server-side handshake processing rather than model inference. A 403 alone does not tell you whether the signature expired; pinning down the cause requires further diagnostic evidence.

A single log line can also record a 403. The value of RUM integration lies in analyzing that result alongside the collected user sessions, app versions, devices, and network environments, so you can judge how widely users are affected and know where to focus regression checks. Aggregating by whatever fields are available also shows whether the failures cluster:

Aggregated pattern	What to check next
More `429/rate-limited` responses	Gateway rate-limiting rules, connection quotas, and attempt volume over the same period
`-1/transport-error` clustering on cellular networks	Client diagnostics, network transitions, and gateway records; the classification alone cannot pinpoint a DNS/TLS root cause
More technical failures on one app version	Network and device distribution under the same criteria, plus version changes and regression results

3. Trace Correlation: Gateway and Authentication Processing

Following that 403, the trace context carried on the handshake resource links to the gateway and authentication spans actually collected. Each span records the duration, status, and call relationships of an operation, which tells you where the rejection happened and whether authentication, routing, or initialization slowed down during the handshake.

If the redacted diagnostic evidence points to an expired signature, troubleshooting then moves to credential refresh, signature validity, and device time, with regression verification against the affected versions or user environments.

Context Propagation and Correlation Prerequisites

The Upgrade request and the RUM resource carry the same traceparent, so both refer to the same context. After the gateway extracts the inbound context, it creates its own span: when the inbound identifiers are trace-id=T, parent-id=A, the gateway uses trace-id=T, span-id=B, parent=A, where A is the parent identifier and B is the gateway's own span ID. The following flow assumes the server side is already instrumented for tracing.

This flow ties the client's handshake result to server-side processing: the client keeps one trace context in both the request and the RUM resource, while the gateway and the authentication service record their processing in their own spans. The context supplies the correlation identifiers; the spans supply processing duration, status, and call relationships. Once server-side tracing and the console-side correlation configuration take effect, you can go from a single handshake record to the exact point of rejection and the processing time behind it.

4. Collection Criteria: Connection Attempts, Final States, and Metrics

The analysis above depends on handshake records collected under a consistent definition. Treating connection lifetime as handshake duration says little about how fast the connection was established, and reporting the same event from multiple callbacks inflates both attempt counts and failure counts. This article therefore treats a single connection attempt as the unit of collection, measures only the time from opening the connection to the first final state, and deduplicates callbacks through one completion function so that each attempt reports at most one handshake resource.

Stop the Timer at the First Final State

This article models the handshake as an HTTP/1.1 Upgrade: timing starts at connect and ends at the first final state — open, a failure, or client-initiated termination — and that interval is one connection attempt. Each attempt reports at most once; a close or failure after open does not add another handshake resource, and a reconnect starts a new attempt. Connection lifetime, message traffic, and AI turns are not counted in this handshake duration.

Cancellation, timeout, and system failure callbacks can arrive one after another or even concurrently. This article funnels final-state events into one completion function so the same attempt is never reported twice. Saving the trigger reason before cancelling the underlying connection lets you separate a user exit from a business timeout: a user exit counts as client-initiated termination and a business timeout counts as a timeout, since the system cancellation callback by itself does not explain why the user ended the connection. The replaced state means only that an unfinished earlier attempt was superseded; it does not account for every reconnect.

Keep the Fields That Support Troubleshooting

Use websocket as the resource type. The core analysis fields are listed below.

Field	Meaning and convention
`url`	Redacted connection address, used for aggregation by Endpoint
`success`	Handshake success confirmed by the `open` callback
`statusCode`	The HTTP status code when one is returned; recorded as `-1` in the example when no response is received
`errorMessage`	Stable, low-cardinality classification such as `auth-rejected`, `timeout`, `cancelled`; raw exception text is not uploaded
`duration`	Milliseconds from `connect` to the first final state, measured with a monotonic clock

Even after receiving a 101, the client can still land in the failure callback because Upgrade validation failed, so this article treats open as confirmation that the handshake succeeded. A value of -1 means no HTTP response was available, which does not in itself indicate a network fault; a WebSocket close code is not written into this field. The error classification is a convention of this article, not a fixed product enumeration.

Group Statistics by Connection Result

When comparing data across versions, networks, and time ranges, this article uses the following criteria:

Success rate: successes divided by the total attempts that reported a final state, keeping cancelled/replaced in the denominator and showing their share separately. If you also compute a technical success rate that excludes deliberate abandonment, exclude only terminations confirmed as deliberate business decisions, and keep timeouts and cancellations of unknown cause.

Duration grouping: compute percentiles separately for successes, failures, and client-initiated terminations, and keep events of unknown cause in their own group. A blended P95 can let fast cancellations mask a slowdown in successful connections.

Data scope: state the collection scope, sampling, and time window. The statistics cover only the records that were reported, not every user attempt.

5. Integration

Take Android's OkHttp and iOS's URLSessionWebSocketTask as examples: the collection logic centers on connection establishment and the final-state callbacks, and both platforms follow the same flow:

Open the connection: start timing this connection attempt, then prepare and inject the trace context before the handshake request goes out.

Handle the final state: handle handshake success, failure, and client-initiated termination in one place, keep the first result, and stop the timer so later callbacks do not report again.

Report the resource: report the connection result, the duration, and the same trace context used in the handshake request through the RUM custom resource API.

Below is the core reporting call after final-state deduplication; the result fields, the duration measurement object, and the context all come from the flow above.

Android:

// resourceUrl is the redacted address; measuring records the duration of this handshake, in ms. 
AlibabaCloudRum.reportCustomResource(
    "websocket", resourceUrl, "GET", statusCode, errorMessage,
    success, "OkHttp3", traceContext, measuring);

iOS:

// resourceURL is the redacted address; measure records the duration of this handshake, in ms.
_ = AlibabaCloudRUM.setCustomResource(
    "websocket", success: success, url: resourceURL, method: "GET",
    statusCode: statusCode, errorMessage: errorMessage,
    provider: "URLSessionWebSocketTask", tracing: traceContext, measure: measure)

Keep your existing business callbacks, message handling, and connection management logic when you integrate. For API usage and configuration details, see the documentation center:

●Android: Custom resources

●iOS: Custom resources

6. Summary

When AI voice interaction on mobile stalls at "Connecting", start troubleshooting at the WebSocket handshake. Record the result and duration of each connection attempt, then combine them with app version, device, and network environment to confirm step by step whether the connection was established and whether failures cluster in one kind of user environment. Together, these details turn the wait into a concrete troubleshooting path.

Integrating custom resources with Cloud Monitor 2.0 RUM links handshake results to user sessions; once client-to-cloud correlation is in place, you can use the trace to inspect the corresponding gateway and authentication processing. RUM gives you the client-side connection result and the user environment, and the trace adds the server-side evidence. Working along the chain of user feedback, handshake record, and server-side processing turns a vague "the AI didn't respond" into a problem with a defined scope and concrete leads.

No More Manual Index Maintenance: SLS Automatically Indexes New Log Fields for Search and Analysis

ObservabilityGuy — Mon, 09 Nov 2026 07:33:41 +0000

This article introduces Alibaba Cloud SLS's field index auto-discovery, which automatically indexes new log fields for immediate search and analysis without manual configuration.

You've probably been here. A production incident hits, you follow the trail, and you finally pin it to a log field that a recent release added — a field carrying exactly the information you need. Then you run the query, and the console tells you: this field has no index configured and cannot be analyzed.

Semi-structured logs are increasingly the norm. The write path lets you add fields freely, but the analysis path still depends on indexes configured in advance. Every release that introduces new fields forces you to patch the index by hand, and covering historical data may mean waiting for a reindex. So the flexibility you get at write time never fully carries over to analysis.

AI Agents make that problem worse. An Agent generates new log fields as it works through a task, and across multiple rounds of inference and retrieval it keeps adjusting its query and analysis path based on intermediate results. Both the fields and the analysis requirements appear at runtime. You can't define them up front, and you can't expect anyone to update the index configuration in real time on every iteration.

Alibaba Cloud Simple Log Service (SLS) now offers field index auto-discovery, which lets indexes keep pace with changing data: SLS creates an index for each newly written field and enables statistics on it automatically, with no manual steps. Developers, operations engineers, and Agents can all analyze new fields without configuring field indexes one by one, so the data is usable out of the box.

1. Manual Index Maintenance Is Struggling to Keep Up with Change

The Field Set Is Rarely Complete at Onboarding

To run log queries and SQL analysis through field indexes, you normally pin down the fields first and then configure the indexes. That works well when the log structure is stable and the query requirements are clear.

In real workloads, however, fields tend to accumulate over time. Application upgrades bring new telemetry, new data sources bring new log structures, and some fields appear only under specific conditions. The log samples you see at onboarding rarely cover everything that arrives later.

Index maintenance therefore becomes an ongoing chore. Each new field raises the same questions: will anyone query it, will anyone analyze it, and is the matching index already in place? The faster the business changes, the harder it is to cover every field.

You Discover the Missing Index Only When You Need the Field

You usually find out whether a field has an index only when you try to use it. A new query needs one more filter condition, or an aggregation needs a different grouping dimension — and the field behind it turns out to have no index.

Adding the index then means waiting before you can analyze. Covering data that has already been written may require a reindex, and the larger the historical volume, the longer the wait.

The data is already there, yet you can't analyze it right away. An investigation that could have gone deeper stalls on index preparation.

Agents Make Log Fields and Query Requirements Even Harder to Predict

When an Agent generates and writes the logs, the log content shifts with the task, the tool calls, and the execution results. Different tasks may record different fields, and execution may add new status values and intermediate results along the way, so the log structure can rarely be pinned down in full at onboarding.

When an Agent analyzes the logs, the analysis path is just as hard to define in advance. The Agent drives the task forward in rounds: it reasons, calls tools, and observes the results. It narrows the scope with field-level search, then uses SQL for aggregation, sorting, and comparison. Each round's results determine which fields to query next and which hypotheses to test, so query requirements emerge dynamically as the reasoning proceeds.

If the Agent hits a field without an index mid-analysis, the query may have to pause while the index is added or rebuilt — or even wait for someone to step in — which breaks the continuous flow of reasoning and tool calling.

2. SLS Field Index Auto-Discovery: New Fields Get Indexes Automatically, No Per-Field Setup

Once field index auto-discovery is enabled, SLS picks up newly written fields that have no index yet, builds indexes for them, and updates the index configuration.

Continuous Discovery: Indexes Built and Statistics Enabled Automatically

Auto-discovery keeps scanning newly written logs. When SLS finds a top-level field with no field index configured, it creates one and enables statistics, so the field can take part in field-specific search and SQL analysis.

Auto-discovery runs continuously alongside log writes. Fields your application adds later, fields introduced by other data sources, and fields that did not exist at onboarding are all discovered when they are actually written, so you no longer patch the configuration after every addition.

Application developers no longer have an extra maintenance step when they add a log field. Data consumers can use the new fields in analysis without extra setup and adjust query conditions and aggregation dimensions as they go.

Indexed on Write: Data Written Before the Configuration Becomes Visible Is Covered Too

With field index auto-discovery enabled, SLS starts building the index for a new field the moment the log is written. Index building and configuration visibility are two separate processes: building starts as soon as the data is written, but the configuration takes some time to sync before it becomes visible.

Data written during that window is still indexed. Once the configuration takes effect, you can query and analyze that data directly, with no need to add or rebuild indexes.

Indexed Uniformly as Text: Handles Fields of Uncertain Type

When a field is discovered automatically, its first observed value is a poor guide to its real business type. A new field might arrive with the value 123 in its first log, which looks numeric, yet later writes could just as easily be abc — a random ID, for example. Committing the index type to numeric on first detection would create a type conflict as soon as a non-numeric character showed up.

SLS therefore indexes every auto-discovered field as text. Text accommodates values in any format, so new fields are written and indexed successfully no matter what they contain, avoiding the type conflicts and missed indexes that inaccurate type inference would cause.

Adaptive SQL Type Conversion: No Type Declaration Needed

Storing every such field as text does not make numeric calculations any harder. During SQL analysis, when the SLS engine encounters a numeric expression on an auto-discovered field, it converts the value to a numeric type at runtime and skips values that cannot be converted. The effect is the same as rewriting sum(x) as sum(try_cast(x as double)).

As a result, in most cases you can write SQL against auto-discovered fields as if they were numeric, with no need to change their type first.

3. How to Enable and Use Field Index Auto-Discovery

Enable Field Index Auto-Discovery in the Console

Go to the index configuration page of the target LogStore, confirm that full-text indexing is enabled, select "Auto-discover unspecified fields (full-text index required)", and save. SLS then starts discovering fields in newly written logs on an ongoing basis.

The console entry point is being rolled out to accounts in batches. If it isn't available to your account yet, submit a ticket to request whitelist access.

View the Auto-Discovered Fields

After you enable the feature, keep writing logs to the LogStore. New fields are indexed automatically as they appear, and you can then view them in the auto-discovered fields section of the index configuration page.

These field indexes all use the text type. They use the same delimiters, case-sensitivity setting, and Chinese-character handling as the full-text index, and statistics are enabled by default.

To change the properties of an auto-discovered field, click the + icon above it to promote it to a manually configured field, then edit its properties.

Use Auto-Discovered Fields for Query and Analysis

Once the field configuration is visible, you can use these auto-discovered fields in LogSearch for field-specific search, quick analysis, and regular SQL analysis, exactly as you would with a manually created field index.

Statistics are enabled by default on auto-discovered fields, so SQL analysis works right away. At the index layer these fields are text, but in SQL you can treat them as numeric.

Enable and Manage Auto-Discovered Fields via SDK

The SDK route works in all regions, with no whitelist needed.

In the index configuration JSON, "auto_key_detect": true in the line field enables field index auto-discovery. auto_text_keys lists the auto-discovered fields.

{
  "line": {
    "auto_key_detect": true,
    "auto_text_keys": ["status", "code", "latency"]
  },
  "keys": {
  }
}

The SDK also exposes APIs for these operations. The following example uses the Python SDK (0.9.52 or later):

from aliyun.log import IndexConfig, IndexLineConfig
#Enable automatic field index discovery
client.create_index(
    project, logstore,
IndexConfig(line_config=IndexLineConfig(
        token_list=[" ", ",", ";", "\n", "\t"],
        auto_key_detect=True,
    ))
)
#Get the configuration related to automatic field index discovery
index = client.get_index_config(project, logstore).get_index_config()
print("Automatic discovery enabled:", index.line_config.auto_key_detect)
print("Discovered fields:", index.line_config.auto_text_keys)

For more configuration details, limits, and FAQs, see the official documentation.

4. Summary

As AI Agents take on task planning and problem analysis, the role of the underlying data system is changing: query requirements can no longer be defined in advance, because they emerge at runtime.

To support this high-frequency, open-ended exploration, a modern data system needs elastic capacity and efficient retrieval for data volumes an LLM cannot consume directly. Just as important, it has to automate the groundwork — field detection and index building — so that analysis isn't held up by wait times or manual intervention.

Field index auto-discovery is one step SLS has taken toward data that adapts on its own. Elastic capacity handles changes in scale, a flexible schema lets the structure evolve, and auto-discovery keeps new fields usable immediately.

When the underlying configuration no longer limits exploration, developers and AI Agents can both focus on the business problem itself. That's what it means for data to be genuinely ready out of the box and open to whatever analysis you need.

Alibaba Cloud SLS Unveils a Major Upgrade to Build the Intelligent Data Engine for the Agent Era

ObservabilityGuy — Mon, 09 Nov 2026 05:32:52 +0000

This article introduces Alibaba Cloud SLS's new Agent-native capabilities, transforming it into an intelligent data engine that provides structured, verifiable, and secure data access for AI agents.

At the 2026 Apsara Conference, Alibaba Cloud Simple Log Service (SLS) announced a major upgrade. On a real-time data foundation that it keeps strengthening, SLS introduced three new Agent-native capabilities: SLS SessionStore for Agent operational data objects; the SLS semantic layer for business context; and, for Agent-native interfaces, the SLS MCP (Model Context Protocol) Server and an orchestration-ready Skills framework. SLS also launched SLS DataAgent, a conversational Agent built for business users. Building on its service to developers, operations staff, and business users, SLS now also supplies real-time data, business context, and standardized tools to Agents, evolving into an intelligent data engine that serves both people and Agents.

1. Agents Enter Production Systems: SLS Builds Agent-native Capabilities on a Real-time Foundation

As Agents move into production systems, the relationship between applications and data is changing. In traditional applications, the users are people and deterministic programs that work through a console or an API; behavior follows fixed workflows with predictable outcomes, and data takes the form of single-line logs, metrics, and trace spans. In Agent applications, the user becomes an Agent that continuously reads data and calls tools; behavior becomes multi-step decision-making and execution, and data expands into sessions and trajectories. An Agent is both a new consumer and a new producer of data, generating operational facts as it uses that data. The value of Agent data also shifts from invocation records to an understanding of the task: the scope extends from the request of a single call to sessions and trajectories that span multiple rounds, models, and tools, reconstructing how a complete task was executed; the depth extends from technical events such as fields, status codes, and response results to business objects, relationships, and rule definitions, as well as task goals, process status, and judgment evidence.

For the Agent era, an intelligent data engine needs four core characteristics: efficient real-time performance, Agent data objects, business semantic context, and Agent-native interfaces. To deliver them, SLS keeps upgrading its real-time data foundation across four stages: real-time ingestion, unified storage and indexing, real-time analysis and computing, and real-time processing. The foundation is real-time, unified, correlatable, and verifiable, and carries general observability data such as logs, metrics, traces, and events; Agent operational data such as messages, tool calls, and steps; and business data such as users, orders, and assets. The new Agent-native capabilities let SLS better supply Agents with trustworthy, understandable data — SLS SessionStore for session-level data organization, the SLS semantic layer for business semantics, and SLS DataAgent for business analysis — and let Agents use SLS more easily and safely, through the SLS MCP Server and a Skills framework for orchestration.

2. SLS SessionStore: Organizing Agent Operational Data into Fact Assets for the First Time

To supply Agents with trustworthy data, SLS first needs to organize how Agents run. Until now, Agent reasoning, model calls, and tool calls have been scattered across traces and logs, with inconsistent fields and events that are hard to pair. A complete task is difficult to reconstruct, let alone evaluate and reuse. The new SLS SessionStore organizes Agent data at the session level. Once an Agent's raw process data is ingested, the system automatically normalizes fields, pairs events, and reconstructs the task process, consolidating it into reusable session fact assets for the first time.

For data organization, SessionStore has a raw fact layer at the bottom that supports two approaches: managed direct write and Ref association with existing LogStores. In the middle, the schema profiler and the mapping profile normalize fields, pair events, and reconstruct the task process. At the top it outputs three semantic views: a trajectory view that reconstructs messages, models, tools, and steps; a session view that shows multi-round interactions and task results; and a feature view that characterizes cost, efficiency, tools, and resources. SessionStore supports multiple data types, including OpenTelemetry (OTel) traces, OTel logs, and LLM Gateway logs. It automatically maps, pairs, and sorts fields, and uses SQL to query, analyze, and reconstruct sessions, trajectories, and raw data, supporting scenarios such as observability, auditing, annotation, and evaluation. Compared with assembling session data from raw traces by hand, SessionStore stands out for zero modification and automation. Existing LogStores can be brought in through Ref association without migration. Field normalization and event pairing happen automatically, and a single pass of organization supports multiple views at once for troubleshooting, evaluation, and cost analysis.

This ensures that every Agent execution is reconstructed into a data asset that can be reviewed, evaluated, and reused, giving downstream observability analysis, quality evaluation, and continuous optimization directly consumable input — without reconstructing the task from raw traces every time.

3. SLS Semantic Layer: Enabling Agents to Understand the Business Behind the Data

Organized data is just the starting point; Agents also need to understand what it means for the business. A general Agent can read fields and values, but it cannot tell what those fields represent in a specific business, which definition a metric should use, or where to set an anomaly threshold. Its answers are therefore prone to drift, and definitions tend to diverge from one place to another. The new SLS semantic layer closes this gap. It automatically extracts a Source Model from operational data and organizes unified semantics by business domain, forming a complete path from SLS data assets through the Source Model and the Context Model to accurate Agent responses.

The path has four steps. First comes the operational fact entry point, which automatically discovers fields, metrics, and terms from SLS data assets, including existing analytical assets such as index configurations, dashboards and alerts, and Scheduled SQL. Second comes the Source Model, where LLMs automatically extract the schema (field definitions), metrics (metric calculation definitions), and terms (business terminology) of every LogStore. Third comes the Context Model, which selects LogStores by business domain, inherits the fact semantics, and adds scenario-specific business context to unify business definitions across projects. Fourth comes Agent integration: SLS DataAgent works out of the box, enterprise-owned Agents connect through APIs, and copilots and tools consume the layer through tool calls. The semantic layer stands out for automation and reuse. The Source Model is extracted automatically by LLMs from real operational data, so nothing has to be defined field by field. The Context Model inherits the fact semantics and adds scenario context, so one modeling effort can be consumed by multiple Agents.

Enterprises do not need to maintain a separate set of business definitions for every Agent. The same question gets a consistent, explainable, traceable, and accurate answer in SLS DataAgent, in enterprise-owned Agents, and in third-party tools.

4. SLS DataAgent: Turning Natural Language Questions into Verifiable Business Conclusions

On top of the semantic layer, SLS DataAgent, launched at the same time, turns SLS's supply-side capabilities directly into analytical conclusions for business users. It offers conversational data query and analysis for business users, runs on SLS real-time operational data, is supported by the Context Model, and operates within enterprise governance. It converts natural language questions into query plans, analysis results, and verifiable conclusions.

DataAgent's differentiator is that it is data-native. General Agents reason well, but when they face enterprise operational data they often cannot judge on their own what a field means in a specific business, which definition a metric uses, how services and resources relate to the business, or what counts as an anomaly. What they lack is not the model but the context. DataAgent grows where the data is generated, turning continuously changing operational data into understandable, trustworthy, real-time context.

Take a game customer service scenario. A player asks, “Why hasn't my top-up arrived?” If an Agent queries the raw logs directly, it faces fragmented fields such as pay_status=success, deliver_status=failed, error_code=BAG_FULL, and item_id=WPN_LEG_001. Correlating across LogStores is complex and business definitions are inconsistent, so the Agent can give only a tentative answer: the delivery may have failed, and neither the cause nor whether a reissue is possible is clear. DataAgent instead queries through the SLS semantic layer, where BAG_FULL has already been translated as “backpack full” and WPN_LEG_001 as “Sky-Rending Blade”. DataAgent can therefore return an evidence-backed business conclusion: payment succeeded, but delivery failed because the player's backpack is full, and once the player frees up space, delivery can be retried according to the rules. By adding business translation between real data and the Agent, the SLS semantic layer turns field values into explainable business conclusions.

Structurally, DataAgent rests on three layers. The perception foundation is rooted in SLS real-time operational data and covers logs, metrics, traces, events, alerts, and resource status. The understanding hub binds to the Context Model to read entities, metric definitions, relationships, anomalies, and verified analysis paths. The action runtime executes tasks within governance, supporting multi-source triggers, dynamic context assembly, tool calls, and tiered auditing. Actions come in three tiers: analysis and suggestions only, execution after human confirmation, and automatic execution under explicit rules and permissions. The Context Model is not a static, one-time artifact: user confirmations, corrections, new queries, and remediation experience keep flowing into context evidence, which after validation and governance incrementally updates the model, so DataAgent becomes more accurate the more it is used. In business scenarios, a DataAgent bound to the relevant context model can continuously report business progress, explain trends, surface opportunities, and flag risks, and can be reused as a shared capability by marketing Agents, operations copilots, and business workflows.

5. Ecosystem Synergy: AgentLoop and STAROps Built on SLS

The SLS real-time data foundation and its new Agent-native capabilities also support agentic applications above it. AgentLoop builds an enterprise Agent quality evaluation and tuning loop on SLS. It ingests Agent operational logs and traces through collection agents (probes) or OpenTelemetry, and SLS handles real-time storage, data transformation, and query and analysis. With SessionStore organizing sessions and reasoning trajectories, AgentLoop keeps accumulating reusable operational data assets and forms a closed loop of observability, evaluation, experimentation, and continuous optimization that covers agentic applications in coding, IT operations, customer service, gaming, and more.

On SLS, STAROps builds a closed loop for full-domain AIOps (artificial intelligence for IT operations). SLS provides the real-time operational data along with the query and analysis that STAROps needs, carrying logs, traces, events, and changes in one place for real-time storage, transformation, and retrieval. UModel then organizes entities, relationship topologies, and data associations to provide traceable context for inference-based diagnosis and recovery validation, supporting natural language queries, alert root cause analysis, change impact analysis, and cross-product fault localization.

For upper-layer applications, SLS provides ready-to-use real-time data along with query and analysis. Neither AgentLoop nor STAROps has to build its own data pipeline; each can focus on evaluation and operations logic. That is why one foundation can support both the quality loop and the operations loop.

6. SLS MCP Server and Skills Framework: Helping Agents Use SLS Better

Beyond powering agentic applications above it, SLS also exposes its capabilities to a wider range of Agents through standard protocols. In the past, calling SLS from an Agent required building an OpenAPI wrapper, managing AccessKeys, and dealing with regions and permissions, which meant a high integration barrier and a real risk of key leakage. The new SLS MCP Server lets Agents use SLS securely through a standard protocol and passwordless OAuth. It logs in with OAuth, runs a local proxy, calls SLS OpenAPI with OAuth credentials on the user's behalf, and controls permissions with Resource Access Management (RAM), so no one has to enter an AccessKey. Sensitive configuration is masked automatically, and the region is matched on access, removing regional restrictions. The MCP Server provides three categories of standard capabilities: resource discovery, log analysis, and alert diagnostics. Compared with wrapping OpenAPI yourself, its advantages are security and zero modification. With OAuth delegated calls, RAM-based permissions, and sensitive configuration masked, any Agent that supports MCP can call SLS securely within its permission boundary without managing keys or changing its integration logic. For developers, integration work that used to be tedious now comes down to a single standard protocol connection.

On top of MCP, SLS also released an orchestration-ready Skills framework. One orchestration Skill (alibabacloud-sls-agent-workflow) serves as the entry point, handling intent recognition, flow orchestration, security gating, and evidence consolidation. It links several specialized Skills that execute in their own domains: LoongCollector collection and operations, index configuration management, query and analysis, intelligent analysis, and SDK guidance. Rather than letting an Agent feel its way through every operation, the Skills framework turns expert experience into standard workflows that can be orchestrated. Users describe a goal in natural language. The orchestration Skill keeps breaking the task down, and each specialized Skill executes in its domain and returns status, evidence, and resource results, ultimately turning the user's goal into an executable, confirmable, and verifiable task.

7. Conclusion

From a real-time data foundation that keeps evolving to the Agent-native capabilities released in this upgrade, SLS has completed a key upgrade for the Agent era. First, the new SessionStore makes Agent operational data organizable and reconstructable; the new semantic layer makes data facts understandable to the business; and the new DataAgent lets business users go from natural language questions straight to verifiable conclusions. Second, the new MCP Server and Skills framework let Agents call SLS securely through standard protocols. Agentic applications such as AgentLoop and STAROps are built on the same foundation. When every piece of operational data has an intelligent Agent that understands it, enterprises building an AI-native data foundation have a real-time, unified, correlatable, and verifiable base to start from.

From Page Rendering to Agents: Unlocking End-to-End Observability for SSR + AI with the ARMS Node.js Instrumentation Agent

ObservabilityGuy — Fri, 06 Nov 2026 02:45:24 +0000

This article introduces the ARMS Node.js agent, which unifies page, model, and service traces for end-to-end SSR and AI observability.

Why is a page taking so long to come back — is rendering slow, is the model slow, or is the business service behind a tool slow? Follow one real request to find where to start looking. (Best suited to developers and site reliability engineers who use server-side rendering (SSR) or are adding AI to Node.js applications.)

Intended for developers and SREs who use SSR or are adding AI to Node.js applications.

"How far along is my refund, and when will the money arrive?"

A user asks a question on the after-sales page and waits for an answer. Once an Agent is in the loop, more work happens behind that page: understanding the question, choosing a tool, querying the order, and turning the result into a reply the user can read.

When the page doesn't come back, where should the team start looking?

If you only look at the entry request's duration, it's hard to tell which layer the wait is on; if you only look at model calls, you may miss the business service behind a tool. The Application Real-Time Monitoring Service (ARMS) Node.js instrumentation agent links server-side rendering, Agent execution, and downstream calls into a single trace, so you start from the page the user visited and follow the request all the way down.

Designed for SSR and AI Agent applications, the Alibaba Cloud ARMS Node.js agent automatically stitches page rendering, model calls, tool execution, and downstream services into one trace. Developers can quickly pinpoint slow requests, wrong parameters, and business bottlenecks, and use logs, metrics, and token usage to reconstruct the full picture — which makes troubleshooting faster.

The Work Behind the Page Has Extended to AI

In an SSR application built with Next.js or Nuxt, page content is generated on the Node.js server. Whatever the rendering process waits for — data, APIs, business logic — affects how fast the page comes back.

When an Agent joins that server-side logic, the page may also wait on model generation, tool queries, and multiple rounds of calls. To the user it's a single page visit; to the developer it's an execution path that runs through the web framework, the AI framework, and business systems.

This is the value of the Node.js instrumentation agent in SSR + AI scenarios: you see both how the page is generated and how the AI Agent completes its task. For teams that build full-stack applications and are adding AI at the same time, both parts can be investigated on the same trace.

The rest of this post walks through a live after-sales query demo. We focus on the server-side trace here; browser loading and interaction can be observed with Real User Monitoring (RUM).

From the After-Sales Page, All the Way Down to the Order Service

In the demo, Next.js serves the after-sales page and Mastra orchestrates the after-sales Agent, which calls qwen-plus through Alibaba Cloud Model Studio. Once the Agent picks the order query tool, the request goes through the Model Context Protocol (MCP) to an MCP server, then calls the order service, reads from and writes to Redis, and finally composes an answer about after-sales progress.

The page waits for the Agent's result before responding. The components are actually running; the order and after-sales content is demo data.

Figure 1 | Flow of the after-sales query (not a console screenshot). The Agent calls a tool to query the downstream order service, builds its answer from the tool result, and hands it back to the page. The blue area marks the server-side trace this post covers.

Open Application Monitoring in Cloud Monitor 2.0, and the execution of the same request unfolds:

Figure 2 | A real demo trace: Next.js SSR → Agent → model and tools → MCP server → order service → Redis. All three applications are correlated by the same trace ID, and this server-side request took about 3.08 seconds.

Drill down from the page entry and you can see the Next.js rendering phase, then keep going to the Agent, model, and tool nodes. Which service the tool called, and whether that service went on to query Redis, are both in this trace.

For a developer, troubleshooting no longer stops at "the after-sales API is slow." The work the page is waiting on now has a location, cross-service calls have context, and you can keep following the same request instead of lining up timestamps across systems first.

What Did the Model Receive, and Which Tools Did It Invoke?

Connecting the trace is one thing — you still need to see what the AI part actually did.

Switch to AI Agent Observability in Cloud Monitor 2.0 and, with the same trace ID, you can see that this after-sales query went through two rounds of model calls: the first turned the question into an order query tool call plus the order number parameter; after the tool returned, the second round produced the refund status, amount, refund method, and expected arrival time.

Figure 3 | Agent view of the same trace. The left side shows the execution steps; the right shows the input to the first model round and the tool call output; the top shows the Agent's final answer, along with token usage and time to first token.

When a user says the answer is wrong, developers can start by checking the question the model actually received, the tool parameters it generated, and the data the tool returned. Whether the model got the right business information is no longer a guess.

When responses get slower, look at model call rounds, time to first token, and tool duration together; when token usage climbs, go back to the specific request and see how it was invoked. Time to first token appears when the framework and the calling mode expose the corresponding streaming data.

Put inputs and outputs, execution steps, and token consumption side by side, and AI troubleshooting finally has a business record you can check against. What you see here is the observable execution path; in production, decide what to collect from user questions, answers, and tool content according to your data security requirements.

If the Tool Is Slow, Drill Down to the Business Service

Here's another request. We added about 1.3 seconds of latency to the order service to simulate an after-sales query waiting on a downstream call.

From the Agent's side, the order query tool took about 1.32 seconds. Drill down through the MCP and the order API took about 1.31 seconds, while the Redis operation took only a few milliseconds.

Figure 4 | A partial trace of another slow request. The demo adds latency to the order service on purpose: the tool call and the order API take similar time, and Redis accounts for only a few milliseconds.

At this point, the next step narrows to what happens inside the order service. Swapping the model or tuning the cache doesn't have to be your first, unverified move; follow the trace, then bring in service logs and code to confirm where the business wait comes from.

Back to the user's question: a page that won't come back doesn't necessarily mean the model is generating slowly. The page waits on the Agent, the Agent waits on the tool, and the tool's wait can be traced further down into the business service.

That's what end-to-end observability buys you: it turns a vague "the AI is slow" into a trail with locations and upstream/downstream relationships.

As the Business Evolves, Observability Doesn't Need to Be Rebuilt Layer by Layer

The after-sales query is just one entry point. The same need shows up in product recommendations, internal knowledge assistants, and any other application that puts AI behind a page.

This demo uses Next.js and Mastra. If your application runs on Nuxt, the agent covers SSR and server-side API observability there too; if your AI is built with Genkit, the agent also collects the execution of flows, models, tools, and retrieval. Tool calls already adapted with the MCP server SDK extend that view further into your business systems.

As pages, workflows, and tools pile up, hand-maintaining instrumentation for each node means continuously maintaining call relationships and collection logic. Framework-level automatic instrumentation cuts down on that repetitive work, and teams can still add business attributes as needed, so observability scales with the business.

Beyond AI, supported dependency calls — HTTP, databases, caches, messaging — and Node.js runtime metrics keep supplying clues on the business side. Traces link to logs to capture the context of a single request, while metrics offer trends across applications, instances, and time windows. Teams can chase one slow request and still reason from the aggregate.

Start with One of Your Pages

If your team runs SSR, or is about to put an Agent into a Node.js application, pick one clear business entry point and let a real request run through the page, the model, and the tools. Then see whether the waits and errors can be traced to a spot on the chain.

Go to the Cloud Monitor 2.0 console and add the ARMS Node.js agent from the Integration Center. Once it's in, check server-side calls and Agent execution under Application Monitoring and AI Agent Observability.

For the exact steps, see Manually Install the Node.js Agent. For supported components and versions, see the official support list.

During integration, make sure the agent loads before your business framework. The "end-to-end" in this post means the server-side trace within the scope of supported components, with context propagating normally and traces retained by sampling. Plugins differ in how they collect, truncate, and mask model and tool inputs and outputs, so check each one, apply the principle of data minimization, and where needed mask the content in advance or turn off collection of that content.

From page rendering to Agent execution, what the user is waiting for is the completion of a business task. Only when that path is clear can developers aim their troubleshooting at the step that actually affects this request.

What's the hardest issue you've had to track down on an SSR or Agent trace? Let us know in the comments.

Friday, 21:47, Checkout Is Slow Again: Twenty Minutes of Troubleshooting with CMS MCP

ObservabilityGuy — Wed, 04 Nov 2026 06:20:36 +0000

This article introduces Alibaba Cloud CMS MCP, which enables AI agents to securely and directly troubleshoot cloud observability issues without local environment setup.

It was 21:47 on a Friday night when Chen's DingTalk went off again.

The alert title was short: checkout service P99 latency above 2 seconds. That service is the core of the checkout path, and if it so much as stutters during the evening peak, the support chat quickly fills up with "Why can't I pay?"

Chen didn't start by opening five consoles. He glanced at Cursor on his screen, which was already connected to the Model Context Protocol (MCP) server for Cloud Monitor (CMS), and gave the Agent a single instruction:

"Find out what just happened to checkout. Get to the root cause, and if monitoring has a gap, fill it."

Twenty minutes later, a conclusion landed in the on-call group: the latency came from the downstream payment gateway, infrastructure events showed no corresponding anomaly, and an error-rate alert had been added.

What changed in those twenty minutes wasn't that the model suddenly "got" observability. It was that, for the first time, the model could investigate, verify, and make changes through the real CMS operating surface — without first assembling a full CLI environment on this particular on-call laptop.

First, Two Things to Get Straight

LLMs are great at reasoning, and at stitching symptoms into hypotheses. What they can't do is get into your Alibaba Cloud account, see the curves in Prometheus, or change an alert rule. The missing piece isn't a longer prompt; it's a set of tools that can be called safely.

MCP is exactly that set of conventions: the model understands the goal and picks the next step, and the tools do the actual work. Think of it as the badge and toolbox you hand the Agent. Without a badge, the model can only analyze from behind the glass. With one, querying alerts, running queries in the Prometheus Query Language (PromQL), pulling a trace tree, and adding a rule all become auditable operations on the cloud.

Alibaba Cloud has already released the official CMS CLI, aliyun cms2. It gathers CMS 2.0 queries and management operations into a single command tree, and that tree was built for Agents from the start: the command contract is stable, output is structured JSON, a failure is reported as a failure, and credentials go through Resource Access Management (RAM) and the Alibaba Cloud configuration chain. You can type the commands in a terminal, scripts and continuous integration (CI) pipelines can call them, and Cursor or Claude Code can run them locally:

aliyun cms2 alert history list --workspace prod-ecommerce
aliyun cms2 metric promql query-range --prometheus-id pi-xxx --query '...'

The cost of the local path isn't the commands themselves; it's the execution environment. The caller has to prepare all of it: the binary installed at a matching version, credentials configured via aliyun configure or environment variables, permission for the process to launch child processes, and network access to the CMS OpenAPI. The on-call engineer's own laptop usually has all of this. Move to a machine without aliyun cms2 installed, or run the Agent in a sandbox with no cloud credentials, and the session breaks off at "command not found" or "not configured."

CMS MCP is not a separate "API for AI." It maps the leaf commands of that same command tree one by one onto MCP tools. alert rule list becomes alert_rule_list; metric promql query-range becomes metric_promql_query_range. The permission model, failure semantics, and audit objects are identical to the CLI. The difference isn't who may use it, but where execution happens: the CLI runs on the Agent's own machine (or in whatever runtime the caller provides), while MCP exposes the same tree at a remote endpoint. Once a client connects, it can call tools/list and tools/call right away; each tool carries its own schema, and no CLI has to be installed on every Agent host.

Last month Chen tried having an Agent call aliyun cms2 locally. It worked on his own machine, but his on-call partner hadn't installed the plugin and used a different profile, so they ended up back in the console. Tonight Cursor is connected to the remote MCP endpoint, the environment lives on the server side, and switching machines mid-shift no longer starts with a round of installation troubleshooting.

21:47: Checkout Hits a Hiccup

The workspace is called prod-ecommerce. Chen has been on call for years, and this alert is nothing new. In the console era, the path ran through the alert list, then Prometheus, then trace search, then back to the rule page. Once the CLI folded those steps into commands, they became precise and repeatable, and an Agent could run them one by one as long as the environment was complete. What really eats time is the exploration itself — you only know which class of tool to reach for next after you've seen the current result — plus the question of whether the execution environment is still in place.

Tonight the environment isn't the problem. He hands the Agent the goal and the permissions and lets the tools follow the evidence.

First, confirm what fired. The Agent calls alert_history_list, filters by workspace, severity, and time, and lands on the rule checkout-p99-latency. Then alert_rule_get pulls the full rule: it watches P99 on the checkout API, fires when the value stays above 2 seconds for 2 consecutive minutes, and notifies the on-call group. At that point the incident is no longer "it timed out again" but a definition you can check against.

Next, check the curves so the alert title doesn't lead you astray. After confirming the Prometheus instance, the Agent calls metric_promql_query_range and pulls the last hour of P99 latency and 5xx error rate. Latency starts climbing at 21:40; the error rate barely moves. The problem is "getting slower," not "starting to fail broadly." Chen has seen too many nights where someone changed code first and only later found that a dependency was simply jittering; this step points the investigation back in the right direction.

Then dig into the traces and catch a slowpoke. trace_search finds spans over 2 seconds for that service name within the time window; one trace ID then goes into trace_tree. The trace tree says more than the curves do:

checkout
 ├─ inventory service     80ms
 ├─ discount calculation  40ms
 └─ payment gateway     2100ms
     └─ third-party channel  2050ms

The root-cause candidates narrow at once: checkout itself isn't slow; the downstream payment call is holding up the entire checkout.

Then cross-check twice:entities and dashboards. entity_query verifies checkout's dependencies in the production environment, so nobody reads the wrong service name or the wrong environment. grafana_dashboard_search finds checkout-overview, and grafana_dashboard_panel_queries extracts the panel queries, which line up with the earlier PromQL. To rule out an unstable machine or network segment, event_hub_list checks cloud service events in the same time window: no matching anomalies in Elastic Compute Service (ECS), Server Load Balancer (SLB), or ApsaraDB RDS.

By now the evidence is solid enough to post to the on-call group. Chen asks for one more thing:

"A latency alert alone isn't enough. Add a rule for checkout 5xx error rate above 1% for 2 consecutive minutes, notifying the same on-call group."

The Agent calls alert_rule_create. If the only change were lowering the original rule's threshold from 2 seconds to 1.5 seconds, the Agent would use alert_rule_patch instead, and any fields left unspecified would keep their existing server-side values. If the on-call group's bot needs double-checking, the Agent can pull the notification channel list at that point.

22:05: the conclusion goes out. No stack of ten screenshots, and no flipping back to the console to double-check.

What Really Matters Here

The core value of CMS MCP isn't that the model writes better PromQL. It's that the most time-consuming stretch of observability work — gathering evidence back and forth across alerts, metrics, traces, entities, dashboards, and events, then turning the result into a single action — now fits inside one session, and execution no longer depends on every Agent machine carrying its own CLI environment.

The aliyun cms2 CLI has already settled the operation contract: the same command can be issued by a person, by an Agent, or by a script or pipeline. When a local Agent goes through the CLI, installation, credentials, network, and version are the caller's responsibility. MCP doesn't change the tree; it changes delivery: remotely discoverable, trimmed by product domain, with schemas that travel with the tools. Troubleshooting isn't ten commands written in advance. You only pull traces once you see that the error rate hasn't moved; you only check entities and dashboards, and rule things out in the Event Center, once the trace tree points at the payment gateway. Sharing a single source with the CLI is what guarantees that the results of that exploration line up with the commands you'd run in a terminal.

This is also why MCP remains bound by RAM. Read-only investigation can run on AliyunCloudMonitorReadOnlyAccess; changing rules or integrations requires write permissions. An on-call identity can be read-only, with changes governed by a separate policy. However capable the Agent is, it shouldn't have access to buttons the operator wouldn't click.

Chen didn't drop the CLI, and he didn't ask every colleague to install the same set of plugins first. He simply stopped spending his most expensive twenty minutes deciding which console to open, or sorting out the local environment.

Core ToolSets

Once MCP maps the leaf commands onto tools, another problem shows up immediately: roughly 191 tools in total. Hand them all to the model at once and the tool descriptions eat the context window; clients also often cap the number of tools. More importantly, a single troubleshooting session has no need for capabilities like "delete workspace" or "create a synthetic monitoring task" sitting in the candidate list.

What a ToolSet mainly does is trim, by product domain, the tool surface the Agent can see and call on a given connection. It isn't a documentation index but a gate on the connection parameters: investigation opens only the core alias by default, build-out and high-risk operations are added to toolsets as needed, and omit_tools can strip out a specific delete-class tool. RAM still decides permissions; the ToolSet decides how wide the menu in front of the model is. That night, Chen went from alert to new rule entirely on the investigation surface. Loading all the tools at once would both crowd out reasoning space and widen the surface for mistaken calls.

The connection parameter defaults to core, which contains metric, meta, trace, alert, event-hub, integration, entity, prometheus, and grafana. Synthetic monitoring, application performance management (APM), workspace creation, and the like are not in the default set. Chen's path from alert to new rule maps exactly to this investigation surface.

ToolSet	What it does during an investigation	Typical tools used that night
`alert`	Query alert history and rules; create, patch, enable, and disable rules	`alert_history_list`, `alert_rule_get`, `alert_rule_create`, `alert_rule_patch`
`metric`	PromQL instant and range queries; basic monitoring for ECS and other cloud services	`metric_promql_query_range`
`meta`	Metric names, units, dimensions, namespaces, and available regions	`meta_metrics`, `meta_namespaces`
`trace`	Search spans, run aggregate queries, and assemble a trace tree by trace ID	`trace_search, trace_tree`
`event-hub`	Query cloud service events and locate the event LogStore	`event_hub_list`
`entity`	Query services, instances, and dependencies; run entity health inspections	`entity_query`
`prometheus`	Prometheus instances, aggregate views, and pre-aggregation rules	`prometheus_instance_get`
`grafana`	Workspaces, data sources, dashboard search, and panel query extraction	`grafana_dashboard_search`, `grafana_dashboard_panel_queries`
`integration`	Integration policies, Addons, collectors, and scrape pre-checks	(Not used that night; relevant only when troubleshooting onboarding)

The ToolSets outside core serve build-out and extension; you don't need to open them for every troubleshooting session:

workspace / datasource / umodel: workspaces, onboarding data sources, entity models
apm / rum: application and frontend observability, instrumentation agent configuration, symbol files
synthetics: site monitoring and detection points
notification-channel: contacts, DingTalk/Lark/WeCom bots, and webhooks
resource-watermark: resource usage reports
aliyun-service / tag / resource-group: activation status, tags, resource groups

Listing the on-call group's bots falls under notification-channel, which isn't part of the default core. With only core open, creating a rule can still reuse the notification recipients from the original rule; to double-check the bot name, set the connection parameter to toolsets=core,notification-channel. Creating a workspace from scratch, onboarding data sources, installing an Addon, configuring APM, and creating synthetic monitoring tasks all work the same way — open the matching ToolSet, with no second protocol to learn.

Beyond the Story: Getting Connected Is Quick

CMS MCP uses the same fixed set of endpoints for all users; simply pick the China site or the international site based on your region. We recommend Streamable HTTP (paths ending in /mcp); use /sse only if your client supports nothing but the older server-sent events (SSE) transport.

China site

Streamable HTTP: https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp

SSE: https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/sse

International site

Streamable HTTP: https://openapi-mcp.ap-southeast-1.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp

SSE: https://openapi-mcp.ap-southeast-1.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/sse

For the default investigation surface, append ?toolsets=core to the URL. When you need to verify the on-call bot, use ?toolsets=core,notification-channel. Authentication uses client OAuth or your existing Alibaba Cloud identity; never place an AccessKey in a configuration file that is committed to Git.

Setup Instructions to Paste into Your Agent

Send the entire block below to Cursor, Claude Code, Lingma, or another Agent, and it will handle the setup for you.

Please add Alibaba Cloud CMS MCP to the current MCP client, without writing the AccessKey into the configuration file.
These are the official endpoints, which are fixed for all users. Pick one according to your network:
- China site Streamable HTTP (recommended): https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp
- China site SSE: https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/sse
- International site Streamable HTTP (recommended): https://openapi-mcp.ap-southeast-1.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp
- International site SSE: https://openapi-mcp.ap-southeast-1.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/sse
Requirements:
1. Prefer Streamable HTTP (/mcp); use SSE (/sse) only if the client does not support Streamable HTTP.
2. Use cms or aliyun-cms as the service name.
3. By default, append ?toolsets=core to the URL; to list alert bots, use ?toolsets=core,notification-channel.
4. Authenticate with OAuth or the existing Alibaba Cloud sign-in session on this machine; never write the AccessKey / Secret into mcp.json.
5. After configuring, call a read-only tool first: list workspaces, or query recent Critical alert history, to confirm connectivity.
For Cursor, write to ~/.cursor/mcp.json (China site sample):
{
  "mcpServers": {
    "aliyun-cms": {
      "url": "https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp?toolsets=core"
    }
  }
}
Claude Code:
claude mcp add --transport http aliyun-cms "https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp?toolsets=core"

Manual Configuration (Cursor)

Write the following JSON into ~/.cursor/mcp.json or the project's .cursor/mcp.json. For the international site, replace the host with openapi-mcp.ap-southeast-1.aliyuncs.com.

{
  "mcpServers": {
    "aliyun-cms": {
      "url": "https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp?toolsets=core"
    }
  }
}

Manual Configuration (Claude Code)

claude mcp add --transport http aliyun-cms \
  "https://openapi-mcp.cn-hangzhou.aliyuncs.com/s/Cms/cms/9fe79ff0bf5748e6923530c5097ec359/mcp?toolsets=core"

Don't Want to Build Your Own Agent? CMS STAROps

MCP suits teams that already run Cursor, Claude Code, or Lingma, or that plan to wire CMS into their own Agent. There's another, more immediate need: no client to install, no mcp.json to write, no local CLI environment to maintain — just open the console, investigate, run inspections, and get a conclusion.

CMS STAROps is built for these users: it provides a managed digital employee rather than one more toolbox you have to wire up yourself. The entry point is in the CMS 2.0 console: select a workspace, then open the artificial intelligence for IT operations (AIOps) assistant. Sessions, skills, permissions, and the execution environment all live in the cloud, and the CLI runs in a sandbox on the console side, so switching machines mid-shift doesn't start with another round of installation.

Three groups of capabilities work out of the box.

Incident investigation. Starting from a single alert or one sentence of natural language, the digital employee correlates metrics, traces, logs, events, and the unified model (UModel) topology to run a quick or deep investigation. Type @ to reference alerts, hosts, applications, pods, and other entities in the workspace, and the analysis expands out from that entity — you don't have to pick tools or write PromQL yourself first. Chen's "find out what happened to checkout" works just as well in STAROps; the conversation simply happens in the console instead of a self-built Agent.
Inspection and diagnosis. Long-running tasks (Missions) support scheduled inspections, such as daily cluster health checks, core service inspections, and operations reports. The output is a structured report you can compare against history, not a one-off chat log. This is a good fit when you need a daily look but don't want to orchestrate the tools yourself every day.
Human-in-the-loop. The digital employee uses its own RAM role, separate from the operator's permissions. "What a person can click" and "what the Agent can call" can be authorized separately. High-risk write operations can require manual confirmation, and conversations, tool calls, and command lines are auditable. This follows the same cloud-side constraints as MCP's "read-only investigation, write operations authorized separately," except that STAROps hosts the Agent and its runtime, rather than requiring the customer to manage their own client.

All three paths can coexist; choose by scenario. Keep using aliyun cms2 in terminals and pipelines. Connect a self-built Agent to CMS MCP. And if you'd rather not build one, use the managed digital employee in CMS STAROps for incident investigation and for inspection and diagnosis. The operating surface is still CMS 2.0; what changes is who owns the Agent.

How the Story Ends

After Chen posted the conclusion, the payments team took the diagnostic report and began investigating the channel latency, and the new monitoring rule took effect the very next minute. That's the work CMS MCP is built to support: not replacing the site reliability engineer (SRE), and not replacing the CLI, but letting an SRE's judgment land immediately on the real CMS operating surface. Let CMS MCP handle your everyday SRE work better: data collection, dashboard design, alert configuration, improving investigations, and inspection reports.

The Player Says 'My Top-Up Never Arrived.' The Log Only Says BAG_FULL.

ObservabilityGuy — Thu, 29 Oct 2026 05:24:09 +0000

This article introduces how the SLS Semantic Layer and DataAgent automate multi-log troubleshooting to deliver verifiable business conclusions.

1. Why Does a Single "Top-Up Not Credited" Complaint Require Checking Multiple Logs?

At 9:47 p.m., support agent Lin picks up an urgent ticket. The player says they just paid 648 CNY, the money is gone, but the top-up package never showed up.

Lin checks the order first: payment succeeded. She checks the payment callback: signature verification passed too. Neither the money nor the payment channel is at fault, so the problem has to be in delivery. She switches to the delivery log and finds a single line: deliver_status = failed. Following the trace_id into the system log, she picks up one more clue: the error code BAG_FULL.

A veteran agent reads this instantly: the player's inventory was full, the system retried delivery, and it kept failing. To someone new to the business, though, this is just a handful of fields and values scattered across different LogStores. To give the player one accurate answer, Lin has to jump back and forth between orders, payments, delivery, and the inventory transaction log, translating machine-recorded facts into a conclusion the player can understand.

This ticket comes from a demo we built, and the logs are synthetic. But cases like this are daily work for game support: top-ups and payments, item changes, matchmaking and ranked play, anti-cheat appeals, social disputes. Answering one of them usually spans several LogStores and takes three to six queries in sequence. The most obvious cost is time: the agent switches repeatedly among orders, payment callbacks, delivery, and the inventory transaction log, several rounds of queries for a single question. And finding the data does not mean understanding it: a newcomer and a veteran can read the same field and reach different conclusions. Add phrases like "revenue," "credited," or "I won but lost points," which different business teams compute differently, and you first have to ask which metric definition the question means.

2. Logs Record Facts, Not Conclusions

The logs are all there. The difficulty is that logs never state the business conclusion outright. Look at deliver_status = delivered and item_id = WPN_LEG_001: neither a person nor an AI can know from the values alone that they mean "delivered" and "Skysplitter Blade." The machine records facts; what those facts mean for the business is not written anywhere in the log.

The SLS semantic layer and DataAgent supply that missing translation. The semantic layer organizes the fields, enum values, business metrics, glossary terms, and samples in your LogStores into a business dictionary. Once the dictionary exists, everyone who asks gets the same metric definition. The DataAgent is the investigator who carries that dictionary. It is attached to at least one Context Model; it maps the player's natural-language question to an explicit metric definition and data source, then builds the queries, analyzes the evidence, and finally produces a conclusion and a draft reply.

3. Three-Layer Collaboration: From Raw Logs to Support Conclusions

All synthetic logs for the sample game are written to the SLS project semantic-game-demo. When troubleshooting, the DataAgent uses three business-evidence LogStores.

LogStore	Main data
game-biz-logstore	Business transaction records for orders, payment callbacks, and delivery; inventory and item changes; gift mail, guild rewards, and season rewards
game-audit-logstore	Audit records for payment risk control, player trades, matchmaking and match settlement, anti-cheat and appeals, reports, and season participation
game-system-logstore	System runtime records for service faults, dependency failures and recovery, performance metrics, and season rule configuration

Most player events carry the common fields user_id, server_id, timestamp, and trace_id. Cross-LogStore troubleshooting then relies on business identifiers such as order_id, battle_id, and item_uuid to string the evidence together. On top of that, each of the three layers has its own job.

(1) Source Model: Describing the Data as It Is

A Source Model maps one-to-one to a project, so you do not name it separately. Once you start automatic generation, the system uses an LLM to analyze dashboards, alerts, Scheduled SQL, and data sampling together, and extracts field semantics, metrics, and glossary terms. Generation time depends on how many LogStores the project holds and how much data they contain; it usually finishes within a few minutes. Extraction results fall into three states by how strong the evidence is. Auto-adopted means the evidence is sufficient and the entry is ready to use. Not adopted means the evidence is thin or ambiguous and someone needs to confirm it. Manually adopted means a person has already checked it. The DataAgent uses only adopted content, so a field meaning still in doubt is never treated as a business fact. A few examples make clear what it extracts.

Semantic element	Game scenario example	Problem it solves
Field	In the order log, deliver_status = delivered means delivered, and amount is denominated in cents	How to interpret fields and enum values
Metric	"Recognized recharge revenue" = SUM(CASE WHEN deliver_status = 'delivered' THEN amount ELSE 0 END)	How a business metric is computed
Glossary term	"Diamonds" = currency_diamond; "Skysplitter Blade" = WPN_LEG_001	How player wording maps to the logs

(2) Context Model: Scoped by Business, Aggregated Centrally

The Context Model is the account-level semantic layer. Each project has one Source Model. When a business spans several projects, you first pick the Source Models of the relevant projects, then select from them the LogStores this business actually needs, and the scattered data semantics come together into one unified model. The game-support Context Model in this article, for instance, selects only the three business-evidence LogStores; unrelated data sources and evaluation-only data sources stay out. Content in a Context Model comes from two places. One is automatic synchronization: whatever the selected LogStores hold in the Source Model flows in automatically, with a maximum delay of 60 seconds, and entries are only added, never removed. The other is manual additions: you can append metrics, glossary terms, and samples, and automatic synchronization never touches these entries. A Context Model aggregated this way serves the DataAgent, and agents such as Codex, OpenClaw, and Claude can connect to it and reuse it as well.

(3) DataAgent: Troubleshooting with a Unified Definition

A DataAgent is a conversational assistant instance, and you attach at least one Context Model when you create it. Given a question, it first identifies the business terms and the metric definition, then chooses data sources, builds queries, strings multi-step log evidence together, and finally outputs a conclusion, the supporting evidence, and a draft reply. You can check every step.

4. Can We Skip the Semantic Layer and Let the LLM Query Logs Directly?

By now you are surely asking: querying SLS logs through Model Context Protocol (MCP) or NL2SQL already works today. If the LLM can write the query itself, why build a semantic layer first?

Take "revenue" from earlier. It could mean order face value, the amount actually paid, or the delivered amount. Ask the LLM directly and it has to guess which field to use and whether to filter for successful delivery; ask the same question twice and it may guess differently. The semantic layer pins down the formula for recognized recharge revenue, so the same metric definition is computed no matter who asks. Now take "Big Sword." That is the name players use; the log stores WPN_LEG_001. Without the glossary mapping, the LLM does not know which log table to open or which value to match, and will likely run a full-text search on the player's wording, find nothing, and tell the player the item was never lost. Then there is that top-up ticket. The useful conclusion is that a full inventory caused delivery to fail, and it takes the payment, delivery, and system logs together to support it. A one-shot query tends to stop at the first plausible answer: it finds deliver_status = failed, blames delivery, and never asks why delivery failed. The DataAgent gets as far as inventory capacity because the Context Model tells it in what order to look and which pieces of evidence to connect.

The query capability is already there. To make queries trustworthy, every conclusion has to map to fixed fields and formulas, and the path of verification has to be deterministic. That is exactly what the semantic layer does. Grant an LLM query permissions without this layer and its answers will read smoothly, so smoothly that you will have trouble noticing whether the metric definition is right and whether the evidence is complete.

5. Best Practice: Build a Game-Support DataAgent in Four Steps

There are three prerequisites. Business logs are already in the project semantic-game-demo, the relevant LogStores have the field indexes required for querying, and the operator has permissions to create and configure a DataAgent.

Step 1: Generate the Source Model for Semantic-game-demo

Open DataAgent from the SLS console home page.

On the source model tab, select the business project semantic-game-demo.

After you start the source model generation task, you can check its status on the details page. Generation time depends on how many LogStores the project holds and how much data they contain; it usually finishes within a few minutes.

When generation finishes, the system lists the field semantics, metrics, and business glossary terms extracted from the LogStores.

For entries that are not adopted, verify the business meaning first, then adopt them in bulk once they are correct, or edit and adopt them one by one.

Step 2: Create the Game Context Model

Create a Context Model and associate the source model of semantic-game-demo. In the description, we recommend stating the business scope the model covers and its main uses, for example support scenarios such as top-ups, items, ranked play, and anti-cheat.

Once associated, the adopted field semantics, metrics, and glossary terms from the source model are synchronized in. You can then add domain-specific terms, metric definitions, and common samples, where a sample can include the question description, the query statement, and the related fields.

Step 3: Create the Game-support DataAgent

Create the DataAgent and attach the context model you built in the previous step.

A DataAgent requires a dedicated RAM role at runtime. By default you can use the system role aliyunslsdataagentrole; if authorization is not yet complete, grant it through the authorization link first. To restrict which resources the agent can access, switch to a custom role, in which case the operator needs the corresponding ram:PassRole permission.

Step 4: Run Conversational Tests with Typical Complaints

Create a conversation, select the DataAgent you just created, and paste in the player's complaint verbatim. Using the context model, the DataAgent identifies the glossary terms and metric definitions, queries the relevant logs, and outputs the key evidence, its analysis, and a draft support reply.

6. How Did the Blind Test Perform?

The test runs on reproducible synthetic logs and covers 13 categories of customer-complaint scenarios.

First, a word on evaluation isolation. game-qa-logstore stores only the questions, the ground truth, and the evaluation evidence; it is not a business query data source. During a blind test, do not grant the DataAgent query permissions on this LogStore, or it will see the ground truth. Take "top-up not credited" as an example. The DataAgent first checks payment: the callback succeeded, signature verification passed, the money really was paid. Next it checks delivery: delivery failed after three retries. It then looks into the cause, and the system error and the inventory capacity point to the same one: a full inventory. The final conclusion rests on the payment, delivery, and system logs together; a single error code cannot get you there.

Now three representative cases.

Player question	Key evidence the DataAgent connects	Conclusion produced
Item missing "Big Sword" WPN_LEG_001	Joins the inventory transaction log with trade records via item_uuid; logon devices normal, no unusual logon; change_reason = trade_sell, sale price 64,663 gold	Based on the available logs, the item was sold on the auction house from the current account, and no evidence of unusual logon or abnormal device activity has been found so far. We recommend providing the trade record and prompting the player to check their account security
Won but lost rating points Match BT_FC3E058B	battle_result = win; rating dropped from 1259 to 1237; rank_change_reason = disconnect_penalty; disconnected for 130–200 seconds, over the 120-second threshold	The rating change comes from the "disconnect counts as forfeit" penalty rule, not from a fault in the normal win-rating logic
Wrongful cheating-ban appeal Event CHT_922FE15A	Detection value 8.85, about 4.1% above the 8.5 threshold; ml_score = 0.62, below the high-confidence bar of 0.7; memory scan result clean; ping 180–250 ms	The available evidence better supports a movement-speed calculation deviation caused by high latency; we recommend handing the case to the anti-cheat team for manual review

The restraint in these conclusions is worth noting. The missing-item conclusion does not say the item "was not stolen"; it says the available logs show no anomalous evidence, and it recommends prompting the player to check their account security. The wrongful-ban appeal does not overturn the ban either; it says the evidence better supports a high-latency deviation and recommends manual review. The conclusion stops where the evidence stops, and the action is left to a person. When a question could map to more than one metric definition, for instance "revenue" meaning order face value, the amount actually paid, or the delivered amount, the DataAgent first asks which definition you want, then continues the query. To reduce these clarifications, add more precise glossary terms, metric descriptions, and samples to the context model.

Making It Work Reliably in Your Own Business

The data is already in SLS, but the knowledge needed to understand it is scattered. Field meanings live in the code, status explanations live in stray documents; dig around and you can still find both. The other two are harder to get. Metric definitions live in business people's heads, and knowing which logs to check for which problem rests on the experience of a few experts. Newcomers keep asking, business people keep depending on engineering, and an AI that does not know the real metric definition can reach a wrong conclusion.

The semantic layer and DataAgent move all this knowledge into one place. Field meanings, metric definitions, and query paths are pinned down; business people can ask in natural language, and the DataAgent understands the question, picks data sources, runs queries, and organizes the evidence, with results you can check. The time newcomers spend asking and engineers spend explaining goes away. You do not have to do it all at once, either: start by getting one question to work end to end. Find the high-frequency question your team answers every day, pick the relevant LogStores, confirm the fields, glossary terms, and metric definitions, add typical phrasings and query samples, and let the DataAgent go from question to conclusion on its own. Once that proves out, extend to more questions, more projects, and more business roles. So we recommend the following:

1. Start with high-frequency, high-value scenarios. Pin down the metric definitions for questions like top-up not credited and duplicate charge first; they make a verifiable baseline easy to establish.

2. Get definitions right before pursuing coverage. Confirm these one by one first: amount units, status enums, time fields, unique identifiers, and metric formulas. If the definitions are wrong, the more questions you cover, the further off you get.

3. Build human review into the process. Set clear boundaries for actions such as refunds, redelivery, bans, and unbans: the DataAgent produces evidence and recommendations, and a person reviews them before execution.

A few limits are worth stating plainly as well:

Limit	Details
Centralized services (context model, DataAgent)	Available only in the cn-beijing central region
Source model	Deployed per region; supported in cn-beijing, cn-shanghai, cn-hangzhou, cn-chengdu, and cn-heyuan
Number of context models/DataAgents	50 each per account and per user by default; you can request an increase

Conclusion: Start with One High-Frequency Question

Back to that opening ticket. With the metric definitions configured, the next time a player asks about a top-up that never arrived, Lin no longer switches back and forth among orders, payments, delivery, and inventory. She hands the question to the DataAgent, gets evidence from three logs plus a draft reply, checks it once, and sends it to the player.

The logs in the demo are synthetic so the case is reproducible. The 13 scenarios are modeled on common customer complaints, not on any one studio's real data. Outside gaming, after-sales, risk control, and O&M have the same kind of question, the kind that takes several logs to answer, and the approach is the same. With your own logs, start with the question your team answers over and over: get the fields, glossary terms, and metric definitions right, make that one work, then decide whether to add more. Questions like this used to be answerable only by the people who understood the logs. Once the first one gets answered, the people who can answer are no longer just those few.

Try the new semantic capabilities in SLS today.

The AI Agent Era Doesn't Need More Data — It Needs a Semantic Layer

ObservabilityGuy — Thu, 29 Oct 2026 02:04:57 +0000

This article introduces UnifiedModel, an open-source semantic layer that organizes enterprise data into a queryable object graph for AI agents.

TL;DR

In the AI Agent era, what's really missing isn't more data — it's an open map of the system.

UModel is an object-graph semantic runtime for enterprise AI. It describes the enterprise world as objects and relationships, and makes those descriptions queryable, verifiable, and callable from agent code. It is not another observability tool, Configuration Management Database (CMDB), or knowledge graph. It sits above them and organizes the facts they already hold into a unified object graph. Taken together, these capabilities mean enterprise data is no longer whatever each system happens to record on its own: it is organized, queried, verified, and invoked around objects.

UnifiedModel is an open source digital twin semantic layer built for Agents. It organizes enterprise assets, runtime data, and system relationships into a queryable object graph. In controlled DataAgentBench experiments, all four flagship models gained 10-20 percentage points once the semantic layer was in place, with GLM-5.2 reaching 50.2% pass@1. Instead of facing isolated data fragments, the Agent reads metrics per object, walks relationships to localize root causes, and actually understands the system it is looking at.

The Problem Isn't a Weak Model — It's the Missing System Structure

Agents have improved fast over the past year. They write code, call tools, plan across multiple steps, and search back and forth across logs, metrics, traces, and change tickets. But drop one into a real production system and ask, "What is actually wrong with payment-gateway right now?" and it will still happily produce an answer that sounds authoritative and is quietly dangerous.

The cause isn't wrong information from any single tool. CPU at 88% is correct. P99 at 2150 ms is correct. That one slow trace is correct too. The problem is that all of these are only symptoms. The Agent has plenty of fragments, but it doesn't know which object each fragment belongs to, how the objects connect, who depends on whom, or which upstream configuration just changed.

This is the engineering version of the blind men and the elephant. Metrics, logs, traces, tickets, and code repositories each touch a real piece of the animal. No single structure tells the Agent what the whole elephant looks like.

The usual first reaction is to give the Agent more capability:

Swap in a stronger model so it can reason longer.
Give it a longer context and push in more logs and documents.
Add retrieval-augmented generation (RAG) so it can retrieve more fragments.
Add memory so it keeps past conversations and lessons.
Wire up more Model Context Protocol (MCP) tools, or even go multi-agent.

Every one of these makes sense, but they add capability and data. They don't address system structure. Connecting to every system is not the same as understanding it. Having every fragment in context is not the same as knowing how they relate.

What's missing is a semantic layer — a structure that captures the objects, relationships, fields, evidence, and action context of the real system.

What a Semantic Layer Really Is: An Externalized World Model

To an Agent, a complex system should not be a pile of tables, APIs, and log files. It should be a queryable world model:

Which objects exist: services, hosts, databases, configurations, deployments, teams, alerts.
How objects connect: services call services, services run on hosts, deployments affect services, configurations act on traces.
Which fields each object class carries: status, owner, service-level objective (SLO), lifecycle, primary key, labels.
Which observability evidence hangs off which object: metrics, logs, traces, events, Runbooks.
Which actions are allowed under which conditions: rollback, rate limiting, scale-out, retry tuning.

In information science, this idea is rooted in ontology — the study of what kinds of things exist in a domain and how they relate. UnifiedModel brings that thinking into engineering systems, describing a digital twin object graph with a very small set of unified primitives.

A semantic layer does not copy all your data into a new database, nor chop documents into segments for a vector store. Think of it as a system map for the Agent. The Agent can still call Prometheus, Simple Log Service (SLS), Elasticsearch (ES), MySQL, Kubernetes, and the CMDB — but it starts from objects rather than from a database engine, follows relationships to the evidence, and then generates an executable query or action plan.

UnifiedModel: Design and Capabilities

The Minimal Primitives of UnifiedModel: Set + Link + Field

UnifiedModel reduces the object graph to three core primitives:

Primitive	Meaning	Typical types
Set	A node — a class of objects or a class of data collections	EntitySet, MetricSet, LogSet, TraceSet, EventSet, RunbookSet
Link	An edge — a relationship among objects, data, and storage	EntitySetLink, DataLink, StorageLink
Field	Field semantics and constraints	Type, unit, semantic role, primary key, mapping, display form

Combine the three and you can express the full entity-data-storage chain:

EntitySet ──DataLink──> DataSet ──StorageLink──> Storage
│
└──EntitySetLink──> EntitySet

Here's the smallest example from an observability scenario:

platform.service is an EntitySet representing the service entity class.
platform.host is another EntitySet, representing hosts.
Services and hosts are joined by a runs_on relationship — that's an EntitySetLink.
A service's latency, error rate, and QPS hang off a MetricSet — that's a DataLink.
That MetricSet physically lives in Prometheus or SLS — that's a StorageLink.
latency_p99_ms is a Field. It carries more than a column name: type, unit, semantics, and query mapping.

What matters here isn't that you drew a graph. It's that the graph can be queried and validated — and discovered and called by an Agent.

Classes and Entities: Define the Model Once, Write to the Runtime Continuously

Two levels in the object graph are easy to conflate: classes and individuals.

Classes are the definition layer — what fields a "service" object has, what its primary key is, which datasets it can associate with, and which entities it can relate to. In UnifiedModel, EntitySet, DataSet, Link, and Field describe this layer. It corresponds to the TBox in ontology.

Individuals are runtime data.checkout-service, payment-gateway, and catalog-api are concrete service entities; payment-gateway calls risk-control is a concrete relationship. They are written, updated, and expired continuously as the system runs. This layer corresponds to the ABox.

A simplified definition layer might look like this:

kind: entity_set
domain: platform
name: platform.service
pk:
  - id
fields:
  - name: id
    type: string
    semantic_role: entity_id
  - name: status
    type: string
  - name: latency_p99_ms
    type: double
    unit: ms

The runtime then writes entities and relationships continuously:

{
  "entity_set": "platform@entity_set@platform.service",
  "entity_id": "payment-gateway",
  "fields": {
    "status": "degraded",
    "latency_p99_ms": 2150
  }
}

This split matters. Classes tell the Agent which types and methods exist in the world; individuals show it what is happening in that world right now. The first is stable, the second dynamic, and only together do they make a live object graph.

Runtime: Turning the Object Graph into One Query Surface for Humans and Agents

UnifiedModel Runtime isn't another isolated platform. It adds a semantic runtime on top of the systems you already run.

Top to bottom, it has four layers:

Access layer: Web UI, CLI, Skills, MCP Gateway. Humans query it; agents query it too.
Runtime services: Workspace, definition validation, entity writes, relationship writes, Query Service, Agent Gateway.
Graph abstraction layer: a single encapsulation of objects, relationships, methods, datasets, and storage mappings.
Storage layer: memory, files, graph databases, plus external data sources such as Prometheus, SLS, ES, and MySQL.

One Query Surface: SPL Covers Definitions, Entities, Topology, and Data Plans

An object graph has to be queryable in a stable way, or it's just documentation. UnifiedModel covers several viewpoints with a single SPL query surface:

Query surface	Purpose
`.umodel`	Query model definitions and metadata
`.entity`	Query specific entities
`.entity_set`	Call methods on an entity class — list datasets, generate a metric plan, generate a log plan
`.topo`	Query topology relationships and neighbors
`.runbook_set`	Query the O&M knowledge and recommended actions attached to an object

An entity query can be very direct:

.entity with(domain='platform', name='platform.service')
| project id, display_name, status, owner
| limit 20

A topology query starts from an object and looks at its neighbors:

.topo
| graph-call getNeighborNodes(
    'platform@entity_set@platform.service',
    'payment-gateway',
    2
  )

Method calls matter even more. An Agent can first ask an EntitySet which methods it has, then call one by signature. A service object, for example, can expose get_metrics, get_logs, and list_data_set. The call doesn't dump full data into the model — it returns an executable plan:

.entity_set with(
  domain='platform',
  name='platform.service',
  ids=['payment-gateway']
)
| entity-call get_metrics('latency_p99_ms', '5m')

The result can be PromQL, an SLS query, ES DSL, or a structured query plan. The Agent executes the plan afterward, which cuts hallucination and reduces the risk of querying the underlying data sources incorrectly.

AI-Friendly: Self-Describing, Progressive Disclosure, MCP, and Skills

"Agent-friendly" doesn't mean stuffing docs into the model. It means making the runtime discoverable.

UnifiedModel does three things for Agents.

First, self-description and progressive disclosure. The Agent doesn't need the full schema up front. It can call __list_method__ to learn which methods the current EntitySet offers, what the parameters are, and what comes back, then decide the next step. Context doesn't fill up all at once, and the Agent never has to guess.

Second, the MCP Gateway. A standard protocol exposes query, explanation, example, and validation capabilities to any Agent. Read tools are available by default; write tools are off by default or require explicit authorization; resources expose metadata only; and every access goes through the Query Service.

Third, Skills. Common capabilities are packaged as loadable skills — object graph queries, root-cause analysis (RCA), impact analysis. Claude Code, Cursor, and Codex then share the same semantic capabilities instead of each Agent writing its own adapter.

The design principle behind all three: discover before calling, get a plan before executing, go through the semantic layer before touching the underlying tools.

UnifiedModel in Practice

Onboarding: From a Real System to a Queryable Object Graph

Onboarding a real system into UnifiedModel takes three steps.

Step 1: model it. Write the model package YAML and define entities, fields, relationships, datasets, and storage mappings. Then run the validation tool to confirm the model is legal.

umctl umodel validate <workspace> --file model-pack.yaml
umctl umodel import <workspace> --file model-pack.yaml

Step 2: write runtime individuals. Entities, relationships, lifecycle, status, and observability evidence stream in continuously. Entities can come from the CMDB, Kubernetes, OpenTelemetry Resource, the Service Catalog, or your deployment system; relationships can come from traces, configuration systems, code dependencies, or manual registration.

umctl entity write <workspace> --file entities.json
umctl topo write <workspace> --file relations.json

Step 3: query the graph. Humans query with .entity and .topo, and Explorer visualizes the result automatically; Agents discover methods through MCP and Skills, generate query plans, and run troubleshooting flows.

Today you write models and data by hand. Automatic registration, OTel mapping, service discovery, and code scanning will keep lowering that cost. The key point: modeling isn't a big-bang project. It grows incrementally — one scenario, one domain, one service path at a time.

Case Study 1: Read Metrics per Object Instead of Hand-Writing PromQL

Back to the question from the talk: someone asks the Agent, "How is payment-gateway doing?"

Without a semantic layer, the Agent has to work out for itself:

Which service object does payment-gateway map to?
Which system holds the metrics?
What is the metric name for P99?
Should the label be service, service_id, or app?
Is the unit seconds or milliseconds?

Get any of these wrong and the result looks reasonable but is useless.

With an object graph, the flow becomes:

Use .entity to locate platform.service/payment-gateway.
Follow DataLinkto the MetricSet and LogSet it's associated with.
Call get_metrics to generate a query plan, with the entity id, window, unit conversion, and metric semantics filled in automatically.

The Agent can then answer:

Status: degraded
QPS: 4200
Error rate: 14.8%
P99: 2150 ms
Replicas: 5/5

The numbers aren't the point. The point is that they were retrieved per object. The Agent didn't need to master a PromQL dialect or guess a label. The object graph turned "that service" into a precise query.

Case Study 2: RCA Isn't Guessing the Latest Release — It's Walking the Relationship Chain

The second scenario is root cause localization: why did payment-gateway blow its P99 SLO?

The object graph gives you a traversable chain of relationships:

Flash Sale 大促
  └─triggers─> cfg-checkout-retry
      └─affects─> checkout-service
          └─calls─> payment-gateway

This chain connects a business event, a configuration change, an upstream service, and the affected service. The Agent no longer just sees that payment-gateway got slow. It walks up the relationships and pins down the cause: the sales promotion triggered the upstream checkout retry configuration, which amplified traffic.

It can also rule out red herrings. A payment-gateway v3.2.1 deployment 12 hours earlier looks suspicious, but the relationships and evidence show it only changed the log format, which doesn't explain a P99 SLO breach. So the Agent doesn't mechanically blame the latest release.

4000 QPS × 3.5x sales promotion traffic × (5 / 2) retry amplification = 35000 QPS 
35000 / 4000 = 8.75x

The conclusion becomes:

The root cause isn't that deployment. It's the upstream retry count going from 2 to 5 and then meeting promotion traffic.
The overload is roughly 8.75× the original capacity.
The recommended action is to roll retries back to 2 and add rate limiting.

Closing: Model the Real World First, Then Organize the Data

In the agent era, what we need to add isn't more fragments. It's a structure that organizes them. Data is just the symptom; objects and relationships are the system structure.

UnifiedModel's core bet is this: use Set + Link + Field to build a real system into an object graph that can be queried, traversed, and executed against; hand that graph to humans and Agents alike through SPL, the Runtime, MCP, and Skills; then close the loop across data, knowledge, and action so the Agent doesn't just see metrics — it understands the system along its real relationships.

One line: model the real world first, then organize the data. The semantic layer is the infrastructure of the Agent era.

Inside STAROps: An Intelligent Log Inspection Architecture for Surfacing Unknown Anomalies

ObservabilityGuy — Fri, 23 Oct 2026 10:21:42 +0000

This article introduces STAROps' AI-driven log inspection architecture for proactively surfacing unknown anomalies.

For most people, log analysis starts with an alert. An API returns errors, so you search for the request ID. A pod restarts, so you look for the exception stack it printed before exiting. A user reports a failed payment, so you take the account and order number and trace the call path. By that point the problem has already happened, and logs help you reconstruct the scene.

But logs hold far more than the scene of a failure. Application logs record how business logic executes, access logs reflect traffic and service quality, audit logs record who did what and when, and Kubernetes Events record changes in resource state. Developers use them to troubleshoot, security teams use them to audit, and operations teams can use them to spot anomalies in payments, inventory, devices, or user behavior.

What gets overlooked are the changes that have not yet triggered an alert and that nobody has thought to search for: a new version starts sporadically surfacing exception stacks no one has seen before; the overall error rate looks fine, yet one region keeps degrading; a class of high-risk operation appears for the first time outside its authorized window.

On their own, these signals look unremarkable. Left alone, they can also become the starting point of the next large-scale outage.

What Intelligent Log Inspection does is take a look when nobody is asking, and tell us: something here has changed in a way it never did before, and it is worth investigating now.

Rules Catch Known Problems; Inspections Look for Unknown Changes

Rules and alerts are useful. We know which error code means failure and which threshold needs attention, so a single SQL query or alerting rule can monitor it indefinitely. These problems are known knowns: we know the problem, and we know how to find it.

Not every problem in a production environment is that clear.

Sometimes we know something is wrong but not where it is concentrated. The error rate climbs from 1% to 10%, and for the moment we cannot say which service, API, version, or tenant is responsible. That is a known unknown.

Other knowledge lives only in engineers' heads. BizException(code=USER_CANCEL) is a normal business cancellation. A 404 on /healthz can be ignored. ci-bot rotating keys in the small hours is an authorized operation. Without that background, the system keeps reporting all of it as problems. Think of this as the unknown known: the team already knows, the system does not.

The hardest case is the unknown unknown: nobody has defined the problem in advance, and nobody knows what to search for. It might be a new log pattern, a combination of fields that has never occurred before, or a long-tail dimension that has just drifted away from normal.

Intelligent Log Inspection cannot promise to find every unknown problem. What it can do is widen the search: alongside the checks already defined, it keeps looking at new patterns, shifts in distribution, numeric tails, and contradictory signals, and surfaces candidates worth investigating as early as possible.

STAROps' Intelligent Log Inspection

When you hand log inspection to an Agent, the hard part is finding anomalies reliably in a huge volume of data while keeping enough evidence for a human to review. Reading every line is impossible, and sampling a handful of lines is unreliable: low-frequency anomalies slip through, and a few dramatic samples can skew the judgment.

STAROps splits the work into several connected stages:

Simple Log Service (SLS) performs broad analysis on the data side first
The LogReduce and dimension drill-down AI operators compress the search space
The Agent plans its investigation path dynamically from the result of each step
Confirmed problems enter a watch list for ongoing tracking
Correlating code, metrics, traces, Skill (a domain skill), and UModel (UnifiedModel, a unified observability modeling framework) extends the investigation across entities and data sources

How SLS Operators and the Agent Divide the Work

In real production, one physical limit cannot be avoided: model context is finite, so you cannot load every log line from an inspection window into it. Naive sampling has its own problems — a low-frequency anomaly may never be sampled, and a few unusual lines can lead the Agent to misjudge the whole picture. Pre-filtering with keywords and fixed SQL only covers the errors you already know about.

SLS provides data-side capabilities such as query and analysis, LogReduce, and dimension drill-down, compressing large volumes of raw logs into structured results a human can review. The STAROps Agent does not replace these capabilities. It generates a plan from the inspection goal, picks time windows, fields, and operators, reviews the results, and then decides whether to drill down further, switch data sources, or stop.

LogReduce extracts stable constants and dynamic variables from indexed text, groups similar logs into patterns (log templates), and then compares pattern counts and variable distributions across windows to find patterns that are new, missing, or spiking. Clustering can also group by fields such as service or module, so the current window can be compared directly against the same period yesterday, the same period last week, or the window before a release.

For example, a mixed set of logs may contain both connection pool exhaustion and downstream call timeouts:

1WARN  [worker-17] Connection pool exhausted, tenant=shop-102, wait=812ms2ERROR [gateway] Downstream timeout, provider=bank-a, api=/pay/confirm, elapsed=3201ms3WARN  [worker-08] Connection pool exhausted, tenant=shop-518, wait=943ms45Pattern A: <*> WARN [<*>] Connection pool exhausted, tenant=<*>, wait=<*>ms6Pattern B: <*> ERROR [gateway] Downstream timeout, provider=<*>, api=<*>, elapsed=<*>ms7

In a real inspection, what the Agent sees is the overall pattern distribution for the analysis window: how often each pattern occurs now and historically, which ones are new or spiking, and how variable values are distributed. "Connection pool exhausted" does not need to be predefined as a keyword — as long as it forms a new log pattern, the inspection still has a chance to surface it.

Dimension drill-down builds a current window and a historical baseline, then searches dimensions such as service, region，version, and operation — and their combinations — for the ranges with the most significant deviation and the largest contribution, reporting the actual value, the baseline value, and the difference.

This works better than running group by on one field after another. The dimension with the most traffic is not necessarily the one causing the anomaly, and a single field may show nothing at all; the real anomaly may appear only in a combination such as project-a + logstore-x. These two operators detect change and narrow scope; neither is a root cause verdict on its own.

Dynamic Planning and Precise Drill-Down

A log inspection does not hard-code one investigation flow for every system. The first time a task runs, STAROps generates an initial plan from the log structure, the available history, and the inspection goal. Stable checks then repeat on that plan, while the live investigation is decided by the Agent from the evidence in the current round.

One investigation might start with a rising failure ratio, confirm the denominator and the historical baseline, and then call the dimension drill-down AI operator to find the range contributing most. Another might start with a new pattern and then check whether it is concentrated in one version, region, or upstream dependency. When fields cannot explain the change, the Agent turns to clustering the message body. When it suspects a release, it checks versions and changes. When it needs to judge user impact, it goes to requests, traces, or business entities.

The first layer of candidates an operator returns is only an entry point. The Agent verifies that the current window and the historical baseline are comparable, re-queries the actual difference for the leading candidates, and then uses the high-contribution ranges as conditions for the next round of queries. It might first pin down project-a, then compare LogStores and versions only within that project. Each step's result determines what the next step queries, the scope narrows layer by layer, and the query chain is preserved in the Finding (a continuously tracked issue record).

Each inspection round also sets aside room for one bounded piece of autonomous exploration. Even when the existing checks report nothing, the Agent picks one direction to keep looking — unexplained logs, new fields, pattern changes, numeric tails, or contradictory signals. It stops when the evidence already explains the problem, when further queries offer no clear gain, or when the data really is insufficient, and it states the conclusion or the data gap explicitly.

Running Findings Continuously and Improving over Time

A problem found in one inspection should not end up as an isolated notification. STAROps creates a Finding that can be tracked over time, holding a stable identity, its current evidence, the scope of impact, and status changes. When the same class of problem appears again, the system links it to the existing Finding and distinguishes whether it is ongoing, getting worse, recurring, or shrinking in impact.

Notification policy also follows how a Finding changes. A new anomaly, a worsening problem, a recurrence, or a clear expansion in scope can trigger a DingTalk or Lark message; unchanged problems stay in the report instead of interrupting the on-call engineer every cycle.

To start an inspection task, you only need to state the business scope, the run cycle, the priorities, and the notification method. Everything after that improves from real feedback. If BizException(code=USER_CANCEL) is a normal business cancellation, tell the Agent. If health checks, stress test traffic, or authorized operations do not need attention, add the applicable scope. STAROps captures confirmed business semantics and the reasoning behind each judgment as Learning, and uses them to adjust later inspections.

Once a human confirms an unknown problem, it also becomes reusable experience. If a given exception stack has been confirmed to relate to connection pool configuration, a later inspection can reuse the previous drill-down path and review method, first checking whether it is the same problem and then whether it is getting worse. When log fields, business scope, or priorities change, the inspection plan is re-evaluated as well; only validated adjustments become stable checks.

Custom Probes and Extensions

Beyond the checks the system generates, you can add your own probe requirements — for example, watching only fund discrepancies inside the settlement window, checking high-risk operations outside authorized hours, or adding dedicated logic for a particular class of business error code. Custom probes join the inspection plan and run alongside LogReduce, dimension drill-down, and the Agent's own exploration.

Skill and Runbook (a standardized troubleshooting manual) add domain knowledge and investigation methods. After a new Java exception stack shows up, the code MCP (Model Context Protocol) server can retrieve the corresponding file, commits, and published version. When a database connection timeout appears, a dedicated troubleshooting Skill can go on to check the connection pool, slow SQL, and database metrics. Enterprises can also write their own business error codes, authorization windows, handling standards, and system knowledge into their own Skills.

When a single LogStore cannot explain a problem, the Agent uses UModel to locate the related entities and where their data lives, then uses Skills to query metrics, traces, changes, or code. UModel organizes entities such as services, pods, nodes, Agent Threads, and Skills together with their LogSets, MetricSets, TraceSets, and actual storage in one relationship graph. The raw data does not have to be moved into a single store; UModel supplies the object relationships, the data locations, and the field semantics.

That keeps a log inspection from stopping at a single LogStore. The general inspection finds the change first, UModel decides which entities and data the investigation can follow, domain Skills decide how each class of object should be queried, and the code MCP server and other tools gather the evidence — forming a chain of evidence: operators surface candidates, the Agent reviews them step by step, and UModel gathers evidence across sources.

How STAROps Uses Intelligent Log Inspection to Improve Business Reliability

For STAROps, healthy service metrics do not mean the business is reliable. A single task passes through the ingress gateway, authentication, AgentRuntime, Sandbox, tool services, and more; a problem at any one of them can make the task fail, slow it down, or return an incomplete result. We also need to keep watching whether tasks complete, whether performance has shifted, and which Agents, users, and dependency paths the errors cluster around.

So we put STAROps' own Agent Thread logs under routine intelligent inspection. The inspection runs hourly and continuously analyzes request volume, completion status, latency, Agent distribution, and user distribution. When it finds a new error pattern or degradation in a key metric, it follows the Agent and UModel to LLM, tool, and gateway data to confirm what the problem affected and where it occurred.

Starting Inspection from a Critical Business Log

Open the target LogStore in the SLS console and go to the intelligent inspection page to create an inspection task. The inspection requirements can stay generic at first, letting the Agent build an initial plan from the characteristics of the logs; if you already have specific business concerns, you can add them at creation time.

After the first run, the inspection found that log record volume had grown 147% over the previous window, with the increase concentrated in one batch task execution period. The report also included the current window, the comparison window, the actual queries, and a note on uncertainty. Based on the available evidence, the growth looked more like a change in system scheduling behavior than anything that directly affected service health.

The result is well-grounded, but still not closely enough aligned with the STAROps business. Record counts and generic fields alone make it hard to tell which entry point and which users the change came from, or whether it affected real tasks.

Adding Business Semantics to the Inspection Plan

We told STAROps directly: skill indicates which entry-point Skill handled the request, employeeId indicates which user the request came from, and both fields deserve close attention. The Agent updated the inspection plan accordingly, adding Skill request distribution, Skill completion status, Skill performance, user request distribution, and a cross analysis of users against Skills — taking the checklist from 6 items to 11. The next inspection round runs these new checks automatically.

The updated inspection showed its value quickly. In one window, overall P90 latency rose from 94 seconds to 171 seconds, which looked like clear performance degradation. When the Agent broke the numbers down by Skill, however, no individual Skill's P90 had degraded; what actually changed was traffic composition, as Skills with a higher share of slow requests accounted for more of the current window. The final report classified it as a change in traffic composition rather than a drop in service performance.

As you can see, user feedback changed how the Agent read the logs, and with it the subsequent checks and drill-down paths. Confirmed business semantics carry into later inspection plans, so nothing has to be explained again each round.

Running Continuously, Notifying on Problems

The inspection task needs to run hourly, but notifications do not need to go out every hour. We added problem trigger conditions to the task: an error rate above 5%, or a new significant finding. STAROps then breaks that down itself: it pushes to the "STAROps Alerts" DingTalk bot only when a new error type appears, when a Skill's non-completion rate is markedly high, when P90 clearly degrades, or when errors concentrate in a small number of users.

In a later inspection, the non-completion rate rose from 1.01% in the previous window to 6.2%, crossing the notification threshold. The notification the bot received carried the error distribution by Skill and by user, performance metrics, and an initial evaluation; the problem had already narrowed to context canceled clustered on one Skill path within a specific time period. Normal inspections that trigger nothing still produce a report, but they do not repeatedly interrupt anyone.

Widening the Investigation with Skills and UModel

A single LogStore can tell us what happened in an Agent Thread, but not necessarily why. We went further and linked this inspection to STAROps' own Skills and UModel: an Agent Thread can be correlated to its entry-point Skill, user, LLM calls, tools, and gateway, and each class of entity has its own LogSet, MetricSet, or other data sources.

When Thread logs show an abnormal non-completion rate, the Agent can follow UModel to check the related entities. In one cross-source inspection, the failure ratio on Threads was still 0%, while the correlated LLM call data showed clustered 429 rate limiting; some gateway SSE statistics and /metrics requests had already been identified as known noise. The final report split the evidence into P1 normal, P2 abnormal, and P3 known noise, rather than lumping every red metric together.

The STAROps reliability inspection began with a single LogStore. The first round was not closely enough aligned with the business, so we added business semantics; the check plan updated accordingly, and the notification policy moved from "send every round" to "notify on problems." Skills and UModel then extended the investigation to more entities and data sources. As the system and the business keep changing, this inspection will keep being calibrated, becoming a continuously running reliability safeguard.

Conclusion

We used to open the logs only after something had gone wrong. Intelligent Log Inspection moves that one step earlier: the Agent continuously watches for pattern and distribution changes in logs, and looks for signals worth investigating even when nobody is asking and no rule exists yet.

A useful inspection task does not need every rule written on day one. It can start from one LogStore and a generic plan, then use its own results to fill in business semantics, adjust the check plan and the notification policy, and finally reach more entities and data sources through Skills and UModel.

Round after round of inspections and feedback turn unknown problems into reusable experience, retire known noise from notifications, and keep tracking the problems that genuinely changed. Problems surface earlier, investigations narrow faster, and every notification that does go out is worth a closer look.

Handling Checkout Peaks with Confidence: How Lemeng Built AIOps for Retail Chains

ObservabilityGuy — Wed, 21 Oct 2026 03:26:15 +0000

This article introduces how Lemeng uses Alibaba Cloud's STAROps to build a proactive AIOps system for retail operations.

Background

When hundreds of thousands of store terminals fire checkout and payment requests at the same time during business peaks, a single moment of jitter anywhere on the path turns into queues at the counter and failed transactions. For Lemeng, a digital services provider for the food retail industry, O&M is not a back-office support function but a front-line business activity that directly shapes the in-store experience and the brand's reputation. Absorbing peak traffic from a vast store network while keeping its multi-tenant SaaS stable through frequent release cycles is the challenge the company has to meet.

Zhejiang Lemeng Information Technology Co., Ltd. ("Lemeng") is a software and information technology service provider focused on digitalization for the food retail industry. Under the Lemeng brand, the company delivers integrated retail digital-intelligence solutions for supermarkets, chain stores and trade distributors, with a product matrix that covers the full range of store operations: a retail POS suite in single-store, chain, supermarket and overseas editions, chain-store ERP, trade distribution and inventory management, a warehouse management system (WMS), a fund and accounting system (aggregated payment and split settlement), omni-channel membership CRM, and a WeChat mini-program new-retail mall, plus extended capabilities such as fresh-produce delivery, visual merchandising and AI smart scales. Its industry solutions cover twelve verticals — supermarkets, snacks, fruit, fresh produce, cooked food, roasted nuts, bakery, tea drinks, catering, meat, alcoholic beverages and trade distribution. To date the company serves more than 200,000 stores in more than 200 cities nationwide, has been chosen by more than 15,000 retail brands and more than 100 leading regional enterprises, and provides 24/7 human support.

Business Challenges

Lemeng's business model makes its O&M far more complex than that of a typical SaaS provider. Huge numbers of store terminals generate checkout and payment requests all at once during business peaks; the product line spans checkout, ERP, WMS, payment and split settlement, mini-programs and more; and observability data sits in different technology stacks. As business lines and store numbers keep growing, that complexity has climbed to a new order of magnitude.

Fragmented alert governance, with real anomalies buried in storms

Checkout SaaS, chain ERP, WMS and payment accounting each relied on a different observability system — Simple Log Service (SLS), Prometheus metrics, application performance management (APM) and cloud service monitoring — so alert rules had to be configured and maintained in separate consoles. As the business expanded, rule counts ballooned. Misconfigurations and omissions were easy to introduce and hard to detect, and traffic peaks during sales promotions and holidays set off alert storms in which engineers on duty struggled to pick out the issues that genuinely needed action.

Observability data hard to produce, hard for the business to consume

Reporting needs across internal R&D, customer success and regional operations teams vary widely, yet all of them matter: API success rates, latency distribution and resource load on the system side; store online rate, checkout transaction volume, payment success rate and split-settlement timeliness on the business side. Building such dashboards usually meant writing complex queries across several kinds of observability data — slow and error-prone. For business users, query syntax was another barrier. Observability stayed in the hands of a few experts, and response times could not keep pace with the business.

Manual monitoring before failures, manual digging after them

On the prevention side, engineers logged in and checked the container clusters and cloud service resources behind the multi-tenant business item by item. There was no reusable inspection SOP and no systematic way to spot resource bottlenecks and latent risks before a failure. On the diagnosis side, a single failed checkout in a store could originate in the store network, the POS terminal, the SaaS application, the database or the payment channel. Cross-system troubleshooting meant switching between platforms again and again and correlating data by hand, and the process leaned heavily on the experience of senior engineers. Every additional minute spent diagnosing the payment path directly affects store operations and financial compliance.

What Lemeng needed was not another monitoring tool, but an artificial intelligence for IT operations (AIOps) system that would unify the alert entry point, lower the barrier to consuming data, move the line of defense forward, and let AI handle cross-system diagnosis.

Solution: Four Layers of AIOps on a Unified Data Foundation

After evaluation, Lemeng chose Alibaba Cloud's Cloud Monitor 2.0 (CMS) together with STAROps, Alibaba Cloud's full-domain AIOps platform, as the core of this upgrade. Lemeng had already accumulated logs, metrics, traces and other observability data on the Alibaba Cloud observability stack, and STAROps uses the UModel observability model to unify them into an end-to-end digital twin that alert governance, automated inspection, root cause analysis and other capabilities then consume. The platform needs no separate activation — it is enabled by default through the STAROps entry point in the CMS 2.0 or SLS console — so it slots smoothly into Lemeng's existing observability setup. The upgrade delivered four layers of capability to address the three challenges.

(1) Unified Alert Governance: Consolidating Alerts from Multiple Systems into a Single Entry Point

With the Alert Management Center in CMS 2.0, Lemeng brought alert rules that had been scattered across SLS, Prometheus metrics, application monitoring and cloud service monitoring into one platform, so engineers no longer jump between systems to configure them separately. Lemeng also brought in the alert rule governance Skill built into STAROps: it inspects existing alert rules automatically and issues governance recommendations, flagging duplicate rules, rules with unsound thresholds and stale rules that have not fired in a long time. Combined with alert and event analysis, it consolidates alerts during peak periods, significantly reducing the drain of alert storms on on-duty efficiency.

(2) Natural-Language Observability: Generating Dashboards with a Single Sentence

Using the natural-language-to-SQL (NL2SQL) capability in STAROps, Lemeng's developers and operations staff describe what they want to observe in plain language, and the platform generates the matching queries and dashboards. Typical uses include checking the trend in checkout API success rates for stores in a given region; comparing latency distribution across payment channels during sales promotions; tallying delay distribution and failure details for split-settlement jobs; and summarizing alert handling and acknowledgment status by line of business. Producing a dashboard has collapsed from a multi-step process — file a request, wait for scheduling, write the query — into a single conversation, so observability data arrives quickly and business roles can use it directly.

(3) Scheduled Intelligent Inspection: Delegating Repetitive Checks to Digital Employees

Through the STAROps long-running task (Mission) mechanism, Lemeng lets digital employees (Agents) run inspections, analyses and change operations on their own, either on a schedule or event-driven, turning repetitive checks that once depended on people into reliable automated flows. The inspection scenarios in production include container cluster health inspection, application health inspection, cloud service resource health inspection, and quality checks on the payment and split-settlement data pipelines. Inspection results are captured as structured reports and can be compared against historical results, so resource bottlenecks and configuration drift surface before they turn into failures. For high-risk actions such as scaling and configuration changes, the human-in-the-loop (HIL) mechanism built into Mission prompts for manual confirmation, raising the level of automation while holding the safety line on changes. This layer upgrades the line of defense from manual item-by-item checking to automated protection.

(4) Dedicated O&M Digital Employee: Rapidly Narrowing Root Causes Across Systems

By combining default rules, Skills and Model Context Protocol (MCP) tools, Lemeng customized the general-purpose digital employee into a dedicated SRE agent fitted to its own business, configuring that agent's responsibilities, permissions and callable tools along its own system boundaries. When an alert fires on the checkout or payment path, the digital employee can trigger rapid root cause analysis from the alert context or from an engineer's natural-language question: using the O&M digital twin built with UModel, it correlates applications, services, resources, topology, alerts and changes in one view, and with AI analysis operators such as metric anomaly detection, log clustering, trace analysis and change retracing, it performs causal inference across systems. Troubleshooting that once relied on senior engineers' experience becomes reusable automated analysis, which directly addresses the problem of long fault-localization chains.

Value Delivered: Three Upgrades from "Reactive Firefighting" to "Proactive Autonomy"

(1) Alert Governance: From Fragmented Manual Work to Unified Standard Actions

Before, each of the four observability systems had its own alert entry point. Rule configuration depended on individual experience, misconfigurations and omissions went unnoticed, alerts flooded the screen at peak times, and engineers on duty had to sift real anomalies out of the noise.

Now, a unified Alert Management Center plus the rule governance Skill turns adding, reviewing and optimizing rules from manual work driven by individual experience into a standard, tool-supported process. Peak-time alerts are consolidated as needed, and alert storms no longer disrupt on-duty judgment. Alerts have been reduced by 70%.

(2) Data Access: From Request-and-Schedule to a Dashboard in One Conversation

Before, a business observability dashboard went through filing a request, waiting for scheduling and writing complex queries — slow to build and error-prone — and business users who did not know the query syntax could only wait for an expert to get back to them.

Now, with NL2SQL and conversational data retrieval, developers, customer success staff and operations teams get core application observability metrics and store-operations data in natural language, without knowing any query syntax. Observability has spread from a handful of experts to a much wider set of business roles, and response times are now measured in minutes.

(3) O&M Posture: From Reactive Response to Proactive Protection, with AI-Assisted Troubleshooting

Before, resource inspection meant checking items by hand, with no guarantee of coverage or consistency, and fault localization meant switching between platforms and correlating data manually, so mean time to repair (MTTR) was hard to bring down consistently.

Now, with a release every two weeks, scheduled Mission inspections keep producing structured reports and comparisons against history, configuration drift is caught early, and automated inspection provides a steady safety net for release quality. When a failure occurs, the dedicated digital employee triggers rapid root cause analysis straight from an alert or a question, and uses the digital twin topology together with AI analysis operators to narrow the search across systems — cutting the time engineers spend switching between consoles and correlating data by hand. MTTR on the checkout and payment paths has been steadily reduced to a matter of minutes.

Future Outlook: Pushing the Boundary of "Proactive Autonomy" Further

Unified alert governance, natural-language observability, automated inspection and intelligent root cause analysis are now running on the core paths. Next, Lemeng and Alibaba Cloud will deepen the work in three directions.

(1) From "Core-Path Pilot" to "Full Business-Line Coverage"

Lemeng will widen the digital employee's remit from the checkout and payment paths to more systems — chain ERP, WMS, mini-program new retail and others — and reuse the proven alert governance, inspection and root cause analysis capabilities as the standard configuration for every product line, so that AIOps becomes standard practice across all business lines.

(2) From "Technical Metric Observability" to "Business Continuity Measurement"

Building on NL2SQL cross-data-source querying, Lemeng will connect technical metrics such as API success rate and latency with business metrics such as store online rate, checkout transaction volume and split-settlement timeliness, building a measurement framework centered on business continuity so that the value of O&M can be expressed directly in business language.

(3) From "Human-Machine Collaboration" to "Higher-Level Autonomous O&M"

Within the safety frame of the human-in-the-loop mechanism, Lemeng will gradually widen the range of scenarios the digital employee closes out on its own, and keep converting the experience gathered in inspection, diagnosis and change into reusable skill assets — so that O&M know-how evolves from individual capability into an organizational asset.

From Reading Bills to Managing Cloud Spend: How STAROps Closes the FinOps Loop

ObservabilityGuy — Mon, 12 Oct 2026 08:23:26 +0000

This article introduces how STAROps enables continuous, AI-driven cloud cost governance and optimization across the FinOps lifecycle.

Once a company is on the cloud, the bill is usually visible enough. What's missing is being able to read it, chase it down, and act on it in time.

Most teams run into the same set of problems:

Finance spots a cost increase only at month-end, and platform and business teams then work backward to find the cause.
Billing fields are complex, and time windows, cost definitions, and usage units are easy to misalign.
Cost shows up only as a total per product, with no way to drill down to billing items, resources, and usage changes.
High or rising spend does not automatically mean waste, and the bill alone cannot tell you whether resource configuration matches actual load.
Inspections rely on manual triggers, so anomaly detection, root-cause analysis, and follow-up lack any continuous mechanism.

FinOps is not about making the bill look better. It is about making cost changes visible in time, explaining them clearly, and routing them back to the right team.

That also means cost governance cannot stop at bill analysis. Even after you pinpoint the product, billing item, and resource behind a cost increase, you have not yet found the optimization opportunity. You still need to look at CPU, memory, disk, connection, bandwidth, and traffic utilization to judge whether resource configuration matches actual load — and then turn that into actionable recommendations under constraints such as business purpose, capacity requirements, and service level agreement (SLA).

STAROps makes this process easy to start and sustain. Users simply ask a question, and the system handles data checks, definition validation, controlled queries, cost drill-down, resource utilization analysis, and optimization recommendations. If the data is not ready yet, it guides users through syncing the bill first, then picks up the analysis where it left off.

From "Reading the Bill" to Managing Cloud Spend

Put these problems into daily work and the picture is familiar.

At month-end the bill gets exported, finance spots a cost spike, the platform team helps break out the details, and engineering confirms whether a business change is behind it. Along the way, everyone keeps re-aligning time windows, product definitions, tag conventions, resource ownership, and cost baselines.

What STAROps adds is the full chain after "seeing the number": align cost and usage definitions, identify abnormal changes, and drill down to specific billing items and resources. Then bring in resource utilization to judge whether configuration matches load, and finally name what to check first, which direction to optimize, and what still needs confirmation. The results can also feed a recurring inspection that keeps tracking cost, usage, utilization, and optimization progress. In STAROps, a cost analysis looks more like this:

From cost changes to resource utilization and optimization recommendations, STAROps extends a single analysis into a cost-governance action you can keep tracking.

Users still ask in natural language, for example:

"Which business line's cloud cost increased most noticeably this month?"
"Simple Log Service (SLS) cost increased in the last 7 days — is it a usage problem or a change in billing item structure?"
"Which resources have no cost-allocation tag, and how does that affect cost attribution?"
"Which billing items had the usage changes that drove the cost increase?"
"Which high-cost resources have stayed at low utilization for a long time, and what optimization actions should be evaluated first?"
"Inspect cost anomalies, cost allocation dimensions, and usage changes for me every day."

What the system really has to answer goes beyond "how much." The bigger questions are: who is using these resources, why cost changed, whether resource configuration matches actual load, which items deserve a closer look, and what optimization direction to take next.

This chain runs the whole way from cost visibility to continuous governance, and it maps onto the three core FinOps phases.

Aligning with the Three FinOps Phases: Inform, Optimize, Operate

Mapped to the FinOps Foundation's three phases — inform, optimize, operate —, STAROps covers the full journey from data reliability and cost visibility to anomaly detection, optimization recommendations, and recurring governance.

FinOps phase	What the enterprise needs to solve	STAROps cost governance capabilities
Inform: see clearly	Are cost, usage, ownership, and billing cycle definitions timely and accurate?	Data ingestion checks, natural language queries, cost attribution, cost-allocation inspection, multi-dimensional reports
Optimize: find opportunities	Which anomalies, waste, and structural changes are worth acting on?	Anomaly filtering, Top N ranking, usage/cost breakdown, resource drill-down, optimization recommendations
Operate: keep it running	How to turn cost governance into a continuous mechanism	Scheduled inspections, parallel tasks, billing data preparation, task recovery, report archiving

These three phases are not a one-time exercise. The cause of today's cost increase can become tomorrow's inspection focus; a cost-allocation gap found this time feeds into the next governance cycle. Each phase reinforces the others in a loop — which is exactly what the FinOps Foundation means by a continuous process.

Inform: Make Cost Data Reliable and Ownership Clear

Reliable Data, Trustworthy Definitions

The first step in FinOps is cost visibility. But "visible" does not mean staring at a single summary table. What actually matters is knowing what data that table covers, how the numbers are calculated, and whether you can break them out by business dimension.

For an enterprise, the more important question is whether these points can be verified:

Can cost be attributed by product, project, department, account, region, and tag?
Can usage and cost be traced to specific resources and billing items?
Can trends, anomalies, and ownership be compared under one consistent set of definitions?
Can business teams understand the cost they own without having to understand complex billing fields?

In the STAROps cost-analysis pipeline, a natural-language question does not turn straight into an ad-hoc SQL query. The system first generates a structured analysis plan — specifying the time window, comparison baseline, metric definitions, aggregation dimensions, filter conditions, sorting rules, and Top N scope — and only then produces a controlled SQL query or request.

To keep results reliable, the system runs two kinds of validation before execution:

Validation type	Purpose
Semantic validation	Checks whether the time window, cost definition, usage units, currency, and dimension combination fit cost analysis semantics
Structural validation	Checks whether query fields, filter conditions, aggregation methods, sorting, and Top N stay within the controlled scope

The upside: users never have to understand the underlying billing fields or write SQL themselves, yet the system still enforces guardrails on query fields, aggregation methods, and cost definitions.

Cost Attribution

User question: Analyze this month's cloud cost by business project and compare cost distribution and changes across projects.

Agent answer (sample data): The e-commerce platform spent ¥100,000 this month, 50% of the total, up ¥20,000 from last month, with the increase coming mainly from Elastic Compute Service (ECS) and ApsaraDB RDS. The data platform spent ¥60,000, or 30%, roughly flat month over month. Internal R&D spent ¥40,000, or 20%, down ¥10,000. Recommendation: have the e-commerce platform team check the newly added compute and database resources first.

FinOps puts a lot of weight on "whoever uses it, owns it." But to make that principle real, cost first has to land on the right dimension.

For cost attribution, STAROps aggregates and filters mainly on fields already in the bill — Tag, ResourceGroup, CostUnit, OwnerID — and lays the cost distribution out for the user:

Tag-based cost allocation: aggregate by the tag field in the bill to view Top N tags, tags with abnormal increases, and the share of cost carried by values like (no tag) or none.
Resource-group cost allocation: aggregate by ResourceGroup to see which resource groups most cost falls into, and how much of it sits in the default resource group.
Cost units: queries and aggregation by CostUnit are supported at the underlying layer, so you can see how cost is distributed across cost units and how much remains unallocated.
Multi-account ownership: split cost by OwnerID across the primary account and financial sub-accounts to establish cost ownership in multi-account setups.

One clarification: without a corporate tagging standard or mapping table, the system will not assert outright that "the cost center is wrong". More precisely, STAROps first aggregates the ownership fields already present in the bill so users can see cost distribution across tag, resource group, cost unit, and account. Whether a given case is a cost allocation gap, a naming inconsistency, or an organizational mapping issue has to be confirmed against the company's own rules.

Looking at this only at month-end leaves you reacting. Put it into a scheduled inspection, and finance spends less time re-confirming numbers at close, while engineering sees the cost impact of new resources and architecture changes much earlier.

Optimize: Find the Opportunities Worth Acting On

Anomaly Detection and Multi-Dimensional Drill-Down

User question: Which ECS cost increases in the last 7 days deserve the most attention? Please drill down to specific instances and billing items.

Agent answer (sample data): ECS is up ¥10,000 (+10%) versus the previous 7 days. On instance i-example03, the "system disk" billing item rose from ¥1,000 to ¥5,000, an increase of ¥4,000 and 40% of the abnormal ECS increase. On instance i-example04, the "cloud server configuration" billing item rose ¥3,000, or 30% of the abnormal increase. Recommendation: check resource changes on these two instances and their billing items first.

A cost increase is not necessarily waste. Business activity, traffic growth, and architecture migration can all drive legitimate growth. What FinOps really needs to watch are abnormal changes that lack an explanation, follow-up, or owner.

STAROps compares cost and usage between the current period and a baseline period, analyzing change amount, change ratio, usage trends, deductions and discounts, resource concentration, and the breakdown across tags, regions, products, and billing items to flag the anomalies worth attention. It also uses the baseline amount to filter out "small-amount, big-percentage" noise — a resource jumping from ¥1 to ¥10 is a steep rise, but usually matters less than a core product line going from ¥100,000 to ¥120,000.

Once anomalies are identified, the system selects the objects that matter by amount and contribution, then drills down level by level:

Drill-down level	Question to answer
Total bill level	Has overall cost deviated from the historical baseline?
Product level	Which cloud products drove most of the change?
Billing item level	Does the change come from storage, traffic, requests, compute, processing, or deductions?
Resource level	Which resources or instances contributed most of the increase?
Cause level	Is the cause usage growth, a unit price change, reduced deductions, or new resources?

STAROps retains the cost change, anomaly contribution, and relative share at every level, so dozens of pages of line items converge into a handful of actionable checks. When SLS cost rises, for example, the analysis breaks it down further into billing items — storage, index traffic, read/write traffic, and data transformation — and cross-references them with specific projects, LogStores, resources, and tags to determine whether the main driver is usage growth or a shift in cost structure. Cost data becomes truly actionable only once you can pin it down to a specific billing item, resource, and cause.

Resource Utilization Analysis and Optimization Recommendations

User question: Analyze ECS cost for the last 7 days and find the low-utilization instances worth checking first.

Agent answer (sample data): Instance i-example01 cost ¥1,000 over 7 days, with CPU averaging 10% and peaking at 20% and memory averaging 10% and peaking at 20% — a low-utilization instance to check first. Instance i-example02 averages 40% CPU with an 80% peak and 60% memory on average, so it is not a low-utilization candidate for now. Optimization advice: first confirm i-example01's business purpose, peak cycle, SLA, and capacity requirements. If the low load is confirmed to be long-term, evaluate downsizing the instance. If it belongs to a test or staging environment that does not run continuously, evaluate scheduled start/stop as well. Before implementing, confirm data protection, dependencies, and the change window.

STAROps analyzes cost change, resource utilization, and business constraints together, taking users from "a cost anomaly was found" to "here is what to check and how to optimize".

The analysis covers:

Cost insight: pinpoint the main cost changes and the products, billing items, and resources behind them.
Utilization analysis: review trends for CPU, memory, disk, connections, bandwidth, and traffic utilization.

Users can read the results from several angles:

Identify the main cost changes and the resources that need attention.
Review resource utilization trends and load behavior.
Get actionable checks and optimization directions that fit the business scenario.
See what still needs business confirmation, so follow-up can be coordinated.

The results explain cost changes, resource utilization, which items to watch, and what still needs business confirmation.

If no Intelligent Utilization Analysis results are available, the report states the information gap explicitly and explains how to enable Intelligent Utilization Analysis.

If additional authorization is required, the report gives clear instructions so the user can complete the analysis.

Operate: Turn FinOps into a Mechanism That Keeps Running

Recurring Inspections and Long-Running Missions

User question: Every day at 9:00 AM, check the previous day's cost anomalies, cost allocation risks, and usage changes, and generate a report.

Agent answer (sample data): Today's inspection is complete: yesterday's total cost was ¥30,000, +10% versus the previous day; 2 key items were found — system disk cost +20%, and cost without an environment tag +10%. The report lists the main contributing resources and the owners to be confirmed. The next inspection will run tomorrow at 9:00 AM.

FinOps cannot rely on a month-end review. Effective cost governance has to keep running day to day.

For recurring cost inspections, STAROps uses its long-running task capability, Mission, to run multiple inspection tracks in parallel:

Inspection area	What it finds
Cost anomalies	Whether total, product, and resource cost fluctuate abnormally
Cost allocation risks	Whether tag, project, department, and account ownership is missing or abnormal
Deduction changes	Compares changes in deducted usage under resource plans, billed usage, and total usage
Structural changes	Identifies new and disappearing products, plus shifts in cost distribution across dimensions such as region and resource group

These tasks run in parallel and then roll up into a single inspection report. The report retains the analysis billing period, baseline period, current amount, baseline amount, change ratio, thresholds triggered, key dimensions, and suggested drill-down directions.

Mission suits exactly this kind of long-running, staged, resumable work: start with the total bill check, then branch out in parallel across product, resource, cost allocation, and usage; once something significant surfaces, keep drilling down.

If an inspection hits temporarily unavailable data, a Cost Manager sync refresh, or an interrupted drill-down, STAROps records the task state and the branches already completed, then resumes once conditions allow. For a production environment, that is far closer to a genuinely long-running system than "fail and rerun by hand."

Preparing Billing Data

Before any cost analysis can run, the billing data has to be queryable.

When a user starts a cost analysis or inspection, STAROps first confirms that Cost Manager has finished syncing the bill. Cost Manager syncs cloud bills into SLS, and every natural language query, structured SQL query, and recurring inspection afterward runs on that billing data.

If the billing data is not ready, the system prompts the user in the STAROps chat to enable Cost Manager, sync billing data, or update the required fields. Once the data is ready, the original analysis continues and the recurring inspection can be triggered again. What the user gets is not "this isn't available right now" but a path forward.

What STAROps Solves for Users

STAROps is not just another billing report. Think of it as a cost-analysis entry point you can question, drill into, and run continuous inspections from.

Common user problem	What STAROps changes
No idea why cost went up	Drills from the total bill down to products, billing items, resources, and usage changes
No appetite for writing SQL or learning complex billing fields	Starts the analysis in natural language; the system generates controlled queries
Worried that analysis definitions are inconsistent	Validates the time window, metric definitions, dimensions, and SQL structure
Too much detail, key items buried	Surfaces the key items using thresholds, Top N ranking, and drill-down results
Cost governance depends on someone remembering to check	Uses Mission to carry recurring inspections and task recovery
Billing data not ready yet	Guides you through enabling Cost Manager and syncing the bill in the STAROps chat, then returns to the analysis

The bottom line: cost questions no longer wait for month-end. You can ask any time, let the system query automatically, and keep inspections running. Cost spikes surface earlier, root causes get explained faster, and governance actions are captured so nothing falls through the cracks.

As enterprise cloud architectures grow increasingly complex, cost governance will shift from after-the-fact analysis to real-time awareness. That is exactly the path STAROps is building — from visible data to explainable anomalies to governance that never stops running.

From Business Alerts to Database Root Causes: How STAROps and ApsaraDB Agent Sharpen Cross-Layer Diagnostics

ObservabilityGuy — Sat, 10 Oct 2026 07:19:30 +0000

This article introduces how STAROps and ApsaraDB Agent enable precise, cross-layer root cause analysis from app alerts to database commands.

The alert fires. The frontend P95 latency spikes to 1,800 ms. You trace the call chain downstream—Redis connections are fluctuating, ApsaraDB RDS (RDS) active sessions are climbing—and the clues gradually converge on the database layer. At this point, opening more monitoring dashboards won't help; what you need is something that can run deep diagnostics against the database itself.

The problem isn't a lack of monitoring data—Simple Log Service (SLS), Application Real-Time Monitoring Service (ARMS), Grafana dashboards—it's all there. What's truly missing is something that can stitch cross-layer data into a causal chain. STAROps excels at full-stack correlation analysis, narrowing the troubleshooting scope from many components down to a few suspects. But once the evidence points to the database layer, you need a database diagnostic partner that can drill down on real metrics. Alibaba Cloud ApsaraDB Agent (product ID: alibabacloud-yaochi-agent) fills exactly that role. In this post, we first explain why cross-layer root cause analysis is hard, then walk through three real-world cases—one RDS and two Redis—escalating from single-instance diagnostics (L3) to cross-layer root cause analysis (L4).

Where Do the Difficulties of Cross-layer Root Cause Analysis Lie?

STAROps integrates metrics, logs, and traces across applications, middleware, and infrastructure for full-stack correlation analysis. That's its strong suit. But once the scope narrows to the database layer, the nature of the problem changes. "Redis anomaly" isn't enough—you need to know whether a Lua script blocked the single thread or a hot key maxed out the CPU. "RDS session spike" isn't enough either—you need to know which table is missing which index and which SQL query is doing a full table scan. These fall squarely in the specialized domain of database diagnostics, beyond what a general observability platform covers. ApsaraDB Agent supports RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB, enabling instance-level diagnostics and root cause analysis based on real metrics. Mount it as a skill on a STAROps digital worker, and a single evidence chain can reach from the application layer all the way down to a specific database command.

A digital worker is an AI operations assistant in STAROps that can be equipped with skills. Attach ApsaraDB Agent to a digital worker and it gains professional diagnostic capabilities at the database layer. ApsaraDB Agent can do far more than fault diagnosis—beyond instance diagnostics, it also covers everyday scenarios such as instance specification selection, kernel parameter change evaluation, cost analysis, schema modeling analysis, kernel parameter explanation, and backup status queries.

Frontend Service Chain and Alert Propagation Path

The cross-layer scenarios that follow all involve the same e-commerce application. Below is the frontend service call chain extracted from Cloud Monitor 2.0, which also represents the actual alert propagation path:

The frontend acts as the entry point for user requests, distributing them via frontend-proxy to business services such as wishlist and promotion. These services in turn call data stores like Redis (Tair, collectively referred to as Redis below) and RDS. When Redis experiences single-thread blocking, the downstream business services are the first to time out. Errors propagate upward layer by layer along the wishlist/promotion → frontend-proxy → frontend path, ultimately surfacing as a slow-response alert on the frontend.

The agent replies shown in the following scenarios are the final diagnostic conclusions. During actual use, the agent first displays analysis progress (querying metrics, analyzing slow commands…). Single-instance diagnostics typically complete within a few minutes, while cross-layer root cause analysis takes roughly 5–10 minutes. If the problem description is incomplete, the agent may ask follow-up questions.

Scenario 1: Abnormal Spike in Active Sessions Due to a Missing Index

Question

Analyze the cause of the abnormal spike in active sessions for RDS instance rm-j6c3l32c730ti37ur during the period 2026-07-18T12:21:30+08:00 → 2026-07-18T12:38:30+08:00

Agent Reply

The digital worker identifies that the issue clearly points to the RDS database layer and invokes ApsaraDB Agent for instance diagnostics.

Within the specified time window, active sessions on the RDS instance rose sharply from a normal level to a peak, lasting about 17 minutes before dropping back. Analysis reveals that during the alert period, a large volume of queries concentrated on the inventory table, all containing correlated subqueries on the product_id column. The client source was IP 10.0.20.210, with 540 concurrent connections executing these queries simultaneously. Because the inventory table lacked an index on product_id, every query performed a full table scan, leading to a backlog of active sessions under high concurrency. Root cause: the inventory table is missing the product_id index, and 540 concurrent high-cost correlated subqueries overwhelmed the RDS instance. Recommendation: add an index on product_id to the inventory table.

Summary

This is an L3 diagnosis—the alert directly points to a single RDS instance with no cross-layer propagation involved. ApsaraDB Agent's diagnostic path: confirm metric anomaly → analyze SQL distribution → trace the client source (IP 10.0.20.210, 540 concurrent connections) → locate the missing index (inventory.product_id). From "elevated active sessions" to "which table is missing which index," the diagnostic precision is enough to directly guide remediation. You can apply the index change right from the RDS console.

Scenario 2: Alert in Frontend, Root Cause in Redis Lua Script

Question

A slow frontend interface response alert occurred from 2026-08-10 10:31 to 10:50. Help me diagnose the root cause of the issue and provide a solution

Agent Reply

The digital worker first kicks off STAROps full-stack correlation analysis, gradually narrowing the troubleshooting scope from the frontend layer to the Redis layer through timeline comparison, topology tracing, and change-record cross-referencing. It then invokes ApsaraDB Agent for in-depth diagnostics at the Redis layer.

The in-depth Redis diagnostics reveal the following key metrics: CPU maxed out at 98 % yet throughput dropped (QPS fell from 1,200 to 180), indicating that an operation was blocking Redis's processing thread. Slow-command log analysis pinpointed the source: a Lua script with heavy computational logic monopolized the Redis single thread, and it fired twice during the alert window. Cascading propagation path: Redis single thread frozen → promotion and wishlist services hit SocketTimeout → frontend-proxy returns timeout errors → frontend response slows down. Root cause: the single thread of the managed Redis instance was monopolized by a Lua slow command with heavy computational logic, firing twice. Recommendation: refactor the Lua script to avoid heavy computation in the script body; consider splitting into pipeline batches.

Summary

This is an L4 diagnosis—the alert fires at the frontend layer, but the root cause lies in the Redis layer, involving cross-layer propagation. The diagnostic path: frontend alert → STAROps full-stack correlation analysis → scope narrowed to Redis → ApsaraDB Agent Redis metric analysis → slow-command log pinpoints Lua script blockage → cascading propagation confirmed (Redis → microservices → frontend). The diagnostic precision goes down to the command type and behavioral characteristics, rather than merely answering "Redis has a performance issue."

Scenario 3: Frontend Response Time Spike, Root Cause in Redis EVAL Command

Question

Analyze the cause of the frontend response time spike alert during the period from 2026-08-10 06:19 to 06:25

Agent Reply

Following the same troubleshooting path as Scenario 2: STAROps full-stack correlation analysis narrows the scope to the Redis layer, and ApsaraDB Agent takes over for in-depth Redis diagnostics.

Redis instance metrics show near-CPU saturation and a sudden QPS drop, a pattern similar to Scenario 2. Slow-command log analysis pinpointed the blockage source: EVAL commands. These commands contain extensive inline logic, executing complex calculations directly within Redis. The single thread is fully occupied during execution, forcing all other commands to queue. Root cause: the Redis single thread is monopolized by slow EVAL commands, causing all commands to queue. Recommendation: locate and block the client source issuing the abnormal EVAL calls, and optimize the script logic to avoid large-scale loops.

Summary

This is also an L4 cross-layer root cause analysis (RCA), with the same "alert in frontend, root cause in Redis" propagation chain and nearly identical metric behavior. Yet the underlying details differ: Scenario 2 involves heavy computation in the business's resident Lua script—fix it by refactoring the script. Scenario 3 involves a temporary inline EVAL script from an anomalous source (EVAL is Redis's command for executing inline Lua scripts)—fix it by blocking the anomalous source and moving the computation out of Redis. Same symptoms, different remediation paths.

Scenario Comparison: Same Symptoms, Different Causes

Looking at Scenario 2 and Scenario 3 together:

	Scenario 2 (10:31 Alert)	Scenario 3 (06:19 Alert)
Symptom	Slow frontend response	Frontend response time spike
Propagation chain	Redis → wishlist/promotion → frontend-proxy → frontend	Same as left
Metric behavior	CPU 98%, QPS 1200→180	CPU saturated, sudden QPS drop
Root cause	Heavy computation in resident business Lua script, monopolizing single thread (occurred twice)	Temporary inline EVAL script monopolizing single thread
Remediation direction	Refactor Lua script	Block anomalous source, move computational logic out of Redis

Same symptom, same middleware, and the root cause mechanism in both cases is "slow commands monopolizing the single thread"—yet the specific causes and remediation actions differ. This is precisely the value of command-level diagnostic precision: it pins the conclusion down to "which command, where it came from, and how to handle it," rather than merely answering "Redis has a performance issue."

Scope

ApsaraDB Agent currently supports the following Alibaba Cloud database products: RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB. Self-managed databases and non-Alibaba Cloud databases are not supported.

When the evidence is insufficient for a definitive conclusion, the agent marks the confidence level and lists potential troubleshooting directions rather than forcing a root cause. Before executing any fix, it is best to have a DBA or engineering lead review the diagnostic conclusions.

Quick Start

Log on to the STAROps console and create a digital employee.
Search for alibabacloud-yaochi-agent under "Skills → Features" and associate it with the employee.
Ask your question in the dialog box. Single-instance diagnostics typically return results within a few minutes; cross-layer root cause analysis takes a bit longer (roughly 5–10 minutes for the scenarios in this post).

You can use the following template to construct your prompt, replacing the content in the brackets:

Analyze the cause of [Anomaly Description] for [Instance Type: RDS/Redis/Tair/MongoDB/PolarDB] instance [Instance ID] during the period from [Start Time] to [End Time]

For example: "Analyze the cause of the abnormal spike in CPU utilization for the Redis instance r-xxxxx during the period from 2026-08-10 10:00 to 11:00." Describe the problem you encountered and the time it occurred using natural language. If you are unsure of the specific instance ID, you can describe the application name and the anomalous symptoms, and the Agent will attempt to correlate it with the underlying database instance.

ApsaraDB Agent raises database-layer diagnostic precision from "some middleware has a problem" to "which specific command or index caused the issue." STAROps, in turn, stitches cross-layer evidence into a complete causal chain. Together, SREs and DBAs can use the same toolset for end-to-end troubleshooting—from alert to fix. For complex faults involving cross-layer propagation, this combination is worth a try.

STAROps currently offers new users 10,000 credits for the first month plus a recurring free tier of 2,500 credits every month. Give STAROps a spin—feed it your most recent production incident and see whether it can actually help: pinpointing the exact command, tracing where it came from, and telling you how to fix it.

Try it now: https://starops.console.aliyun.com/