DEV Community: ObservabilityGuy

Inside STAROps: An Intelligent Log Inspection Architecture for Surfacing Unknown Anomalies

ObservabilityGuy — Fri, 23 Oct 2026 10:21:42 +0000

This article introduces STAROps' AI-driven log inspection architecture for proactively surfacing unknown anomalies.

For most people, log analysis starts with an alert. An API returns errors, so you search for the request ID. A pod restarts, so you look for the exception stack it printed before exiting. A user reports a failed payment, so you take the account and order number and trace the call path. By that point the problem has already happened, and logs help you reconstruct the scene.

But logs hold far more than the scene of a failure. Application logs record how business logic executes, access logs reflect traffic and service quality, audit logs record who did what and when, and Kubernetes Events record changes in resource state. Developers use them to troubleshoot, security teams use them to audit, and operations teams can use them to spot anomalies in payments, inventory, devices, or user behavior.

What gets overlooked are the changes that have not yet triggered an alert and that nobody has thought to search for: a new version starts sporadically surfacing exception stacks no one has seen before; the overall error rate looks fine, yet one region keeps degrading; a class of high-risk operation appears for the first time outside its authorized window.

On their own, these signals look unremarkable. Left alone, they can also become the starting point of the next large-scale outage.

What Intelligent Log Inspection does is take a look when nobody is asking, and tell us: something here has changed in a way it never did before, and it is worth investigating now.

Rules Catch Known Problems; Inspections Look for Unknown Changes

Rules and alerts are useful. We know which error code means failure and which threshold needs attention, so a single SQL query or alerting rule can monitor it indefinitely. These problems are known knowns: we know the problem, and we know how to find it.

Not every problem in a production environment is that clear.

Sometimes we know something is wrong but not where it is concentrated. The error rate climbs from 1% to 10%, and for the moment we cannot say which service, API, version, or tenant is responsible. That is a known unknown.

Other knowledge lives only in engineers' heads. BizException(code=USER_CANCEL) is a normal business cancellation. A 404 on /healthz can be ignored. ci-bot rotating keys in the small hours is an authorized operation. Without that background, the system keeps reporting all of it as problems. Think of this as the unknown known: the team already knows, the system does not.

The hardest case is the unknown unknown: nobody has defined the problem in advance, and nobody knows what to search for. It might be a new log pattern, a combination of fields that has never occurred before, or a long-tail dimension that has just drifted away from normal.

Intelligent Log Inspection cannot promise to find every unknown problem. What it can do is widen the search: alongside the checks already defined, it keeps looking at new patterns, shifts in distribution, numeric tails, and contradictory signals, and surfaces candidates worth investigating as early as possible.

STAROps' Intelligent Log Inspection

When you hand log inspection to an Agent, the hard part is finding anomalies reliably in a huge volume of data while keeping enough evidence for a human to review. Reading every line is impossible, and sampling a handful of lines is unreliable: low-frequency anomalies slip through, and a few dramatic samples can skew the judgment.

STAROps splits the work into several connected stages:

Simple Log Service (SLS) performs broad analysis on the data side first
The LogReduce and dimension drill-down AI operators compress the search space
The Agent plans its investigation path dynamically from the result of each step
Confirmed problems enter a watch list for ongoing tracking
Correlating code, metrics, traces, Skill (a domain skill), and UModel (UnifiedModel, a unified observability modeling framework) extends the investigation across entities and data sources

How SLS Operators and the Agent Divide the Work

In real production, one physical limit cannot be avoided: model context is finite, so you cannot load every log line from an inspection window into it. Naive sampling has its own problems — a low-frequency anomaly may never be sampled, and a few unusual lines can lead the Agent to misjudge the whole picture. Pre-filtering with keywords and fixed SQL only covers the errors you already know about.

SLS provides data-side capabilities such as query and analysis, LogReduce, and dimension drill-down, compressing large volumes of raw logs into structured results a human can review. The STAROps Agent does not replace these capabilities. It generates a plan from the inspection goal, picks time windows, fields, and operators, reviews the results, and then decides whether to drill down further, switch data sources, or stop.

LogReduce extracts stable constants and dynamic variables from indexed text, groups similar logs into patterns (log templates), and then compares pattern counts and variable distributions across windows to find patterns that are new, missing, or spiking. Clustering can also group by fields such as service or module, so the current window can be compared directly against the same period yesterday, the same period last week, or the window before a release.

For example, a mixed set of logs may contain both connection pool exhaustion and downstream call timeouts:

1WARN  [worker-17] Connection pool exhausted, tenant=shop-102, wait=812ms2ERROR [gateway] Downstream timeout, provider=bank-a, api=/pay/confirm, elapsed=3201ms3WARN  [worker-08] Connection pool exhausted, tenant=shop-518, wait=943ms45Pattern A: <*> WARN [<*>] Connection pool exhausted, tenant=<*>, wait=<*>ms6Pattern B: <*> ERROR [gateway] Downstream timeout, provider=<*>, api=<*>, elapsed=<*>ms7

In a real inspection, what the Agent sees is the overall pattern distribution for the analysis window: how often each pattern occurs now and historically, which ones are new or spiking, and how variable values are distributed. "Connection pool exhausted" does not need to be predefined as a keyword — as long as it forms a new log pattern, the inspection still has a chance to surface it.

Dimension drill-down builds a current window and a historical baseline, then searches dimensions such as service, region，version, and operation — and their combinations — for the ranges with the most significant deviation and the largest contribution, reporting the actual value, the baseline value, and the difference.

This works better than running group by on one field after another. The dimension with the most traffic is not necessarily the one causing the anomaly, and a single field may show nothing at all; the real anomaly may appear only in a combination such as project-a + logstore-x. These two operators detect change and narrow scope; neither is a root cause verdict on its own.

Dynamic Planning and Precise Drill-Down

A log inspection does not hard-code one investigation flow for every system. The first time a task runs, STAROps generates an initial plan from the log structure, the available history, and the inspection goal. Stable checks then repeat on that plan, while the live investigation is decided by the Agent from the evidence in the current round.

One investigation might start with a rising failure ratio, confirm the denominator and the historical baseline, and then call the dimension drill-down AI operator to find the range contributing most. Another might start with a new pattern and then check whether it is concentrated in one version, region, or upstream dependency. When fields cannot explain the change, the Agent turns to clustering the message body. When it suspects a release, it checks versions and changes. When it needs to judge user impact, it goes to requests, traces, or business entities.

The first layer of candidates an operator returns is only an entry point. The Agent verifies that the current window and the historical baseline are comparable, re-queries the actual difference for the leading candidates, and then uses the high-contribution ranges as conditions for the next round of queries. It might first pin down project-a, then compare LogStores and versions only within that project. Each step's result determines what the next step queries, the scope narrows layer by layer, and the query chain is preserved in the Finding (a continuously tracked issue record).

Each inspection round also sets aside room for one bounded piece of autonomous exploration. Even when the existing checks report nothing, the Agent picks one direction to keep looking — unexplained logs, new fields, pattern changes, numeric tails, or contradictory signals. It stops when the evidence already explains the problem, when further queries offer no clear gain, or when the data really is insufficient, and it states the conclusion or the data gap explicitly.

Running Findings Continuously and Improving over Time

A problem found in one inspection should not end up as an isolated notification. STAROps creates a Finding that can be tracked over time, holding a stable identity, its current evidence, the scope of impact, and status changes. When the same class of problem appears again, the system links it to the existing Finding and distinguishes whether it is ongoing, getting worse, recurring, or shrinking in impact.

Notification policy also follows how a Finding changes. A new anomaly, a worsening problem, a recurrence, or a clear expansion in scope can trigger a DingTalk or Lark message; unchanged problems stay in the report instead of interrupting the on-call engineer every cycle.

To start an inspection task, you only need to state the business scope, the run cycle, the priorities, and the notification method. Everything after that improves from real feedback. If BizException(code=USER_CANCEL) is a normal business cancellation, tell the Agent. If health checks, stress test traffic, or authorized operations do not need attention, add the applicable scope. STAROps captures confirmed business semantics and the reasoning behind each judgment as Learning, and uses them to adjust later inspections.

Once a human confirms an unknown problem, it also becomes reusable experience. If a given exception stack has been confirmed to relate to connection pool configuration, a later inspection can reuse the previous drill-down path and review method, first checking whether it is the same problem and then whether it is getting worse. When log fields, business scope, or priorities change, the inspection plan is re-evaluated as well; only validated adjustments become stable checks.

Custom Probes and Extensions

Beyond the checks the system generates, you can add your own probe requirements — for example, watching only fund discrepancies inside the settlement window, checking high-risk operations outside authorized hours, or adding dedicated logic for a particular class of business error code. Custom probes join the inspection plan and run alongside LogReduce, dimension drill-down, and the Agent's own exploration.

Skill and Runbook (a standardized troubleshooting manual) add domain knowledge and investigation methods. After a new Java exception stack shows up, the code MCP (Model Context Protocol) server can retrieve the corresponding file, commits, and published version. When a database connection timeout appears, a dedicated troubleshooting Skill can go on to check the connection pool, slow SQL, and database metrics. Enterprises can also write their own business error codes, authorization windows, handling standards, and system knowledge into their own Skills.

When a single LogStore cannot explain a problem, the Agent uses UModel to locate the related entities and where their data lives, then uses Skills to query metrics, traces, changes, or code. UModel organizes entities such as services, pods, nodes, Agent Threads, and Skills together with their LogSets, MetricSets, TraceSets, and actual storage in one relationship graph. The raw data does not have to be moved into a single store; UModel supplies the object relationships, the data locations, and the field semantics.

That keeps a log inspection from stopping at a single LogStore. The general inspection finds the change first, UModel decides which entities and data the investigation can follow, domain Skills decide how each class of object should be queried, and the code MCP server and other tools gather the evidence — forming a chain of evidence: operators surface candidates, the Agent reviews them step by step, and UModel gathers evidence across sources.

How STAROps Uses Intelligent Log Inspection to Improve Business Reliability

For STAROps, healthy service metrics do not mean the business is reliable. A single task passes through the ingress gateway, authentication, AgentRuntime, Sandbox, tool services, and more; a problem at any one of them can make the task fail, slow it down, or return an incomplete result. We also need to keep watching whether tasks complete, whether performance has shifted, and which Agents, users, and dependency paths the errors cluster around.

So we put STAROps' own Agent Thread logs under routine intelligent inspection. The inspection runs hourly and continuously analyzes request volume, completion status, latency, Agent distribution, and user distribution. When it finds a new error pattern or degradation in a key metric, it follows the Agent and UModel to LLM, tool, and gateway data to confirm what the problem affected and where it occurred.

Starting Inspection from a Critical Business Log

Open the target LogStore in the SLS console and go to the intelligent inspection page to create an inspection task. The inspection requirements can stay generic at first, letting the Agent build an initial plan from the characteristics of the logs; if you already have specific business concerns, you can add them at creation time.

After the first run, the inspection found that log record volume had grown 147% over the previous window, with the increase concentrated in one batch task execution period. The report also included the current window, the comparison window, the actual queries, and a note on uncertainty. Based on the available evidence, the growth looked more like a change in system scheduling behavior than anything that directly affected service health.

The result is well-grounded, but still not closely enough aligned with the STAROps business. Record counts and generic fields alone make it hard to tell which entry point and which users the change came from, or whether it affected real tasks.

Adding Business Semantics to the Inspection Plan

We told STAROps directly: skill indicates which entry-point Skill handled the request, employeeId indicates which user the request came from, and both fields deserve close attention. The Agent updated the inspection plan accordingly, adding Skill request distribution, Skill completion status, Skill performance, user request distribution, and a cross analysis of users against Skills — taking the checklist from 6 items to 11. The next inspection round runs these new checks automatically.

The updated inspection showed its value quickly. In one window, overall P90 latency rose from 94 seconds to 171 seconds, which looked like clear performance degradation. When the Agent broke the numbers down by Skill, however, no individual Skill's P90 had degraded; what actually changed was traffic composition, as Skills with a higher share of slow requests accounted for more of the current window. The final report classified it as a change in traffic composition rather than a drop in service performance.

As you can see, user feedback changed how the Agent read the logs, and with it the subsequent checks and drill-down paths. Confirmed business semantics carry into later inspection plans, so nothing has to be explained again each round.

Running Continuously, Notifying on Problems

The inspection task needs to run hourly, but notifications do not need to go out every hour. We added problem trigger conditions to the task: an error rate above 5%, or a new significant finding. STAROps then breaks that down itself: it pushes to the "STAROps Alerts" DingTalk bot only when a new error type appears, when a Skill's non-completion rate is markedly high, when P90 clearly degrades, or when errors concentrate in a small number of users.

In a later inspection, the non-completion rate rose from 1.01% in the previous window to 6.2%, crossing the notification threshold. The notification the bot received carried the error distribution by Skill and by user, performance metrics, and an initial evaluation; the problem had already narrowed to context canceled clustered on one Skill path within a specific time period. Normal inspections that trigger nothing still produce a report, but they do not repeatedly interrupt anyone.

Widening the Investigation with Skills and UModel

A single LogStore can tell us what happened in an Agent Thread, but not necessarily why. We went further and linked this inspection to STAROps' own Skills and UModel: an Agent Thread can be correlated to its entry-point Skill, user, LLM calls, tools, and gateway, and each class of entity has its own LogSet, MetricSet, or other data sources.

When Thread logs show an abnormal non-completion rate, the Agent can follow UModel to check the related entities. In one cross-source inspection, the failure ratio on Threads was still 0%, while the correlated LLM call data showed clustered 429 rate limiting; some gateway SSE statistics and /metrics requests had already been identified as known noise. The final report split the evidence into P1 normal, P2 abnormal, and P3 known noise, rather than lumping every red metric together.

The STAROps reliability inspection began with a single LogStore. The first round was not closely enough aligned with the business, so we added business semantics; the check plan updated accordingly, and the notification policy moved from "send every round" to "notify on problems." Skills and UModel then extended the investigation to more entities and data sources. As the system and the business keep changing, this inspection will keep being calibrated, becoming a continuously running reliability safeguard.

Conclusion

We used to open the logs only after something had gone wrong. Intelligent Log Inspection moves that one step earlier: the Agent continuously watches for pattern and distribution changes in logs, and looks for signals worth investigating even when nobody is asking and no rule exists yet.

A useful inspection task does not need every rule written on day one. It can start from one LogStore and a generic plan, then use its own results to fill in business semantics, adjust the check plan and the notification policy, and finally reach more entities and data sources through Skills and UModel.

Round after round of inspections and feedback turn unknown problems into reusable experience, retire known noise from notifications, and keep tracking the problems that genuinely changed. Problems surface earlier, investigations narrow faster, and every notification that does go out is worth a closer look.

Handling Checkout Peaks with Confidence: How Lemeng Built AIOps for Retail Chains

ObservabilityGuy — Wed, 21 Oct 2026 03:26:15 +0000

This article introduces how Lemeng uses Alibaba Cloud's STAROps to build a proactive AIOps system for retail operations.

Background

When hundreds of thousands of store terminals fire checkout and payment requests at the same time during business peaks, a single moment of jitter anywhere on the path turns into queues at the counter and failed transactions. For Lemeng, a digital services provider for the food retail industry, O&M is not a back-office support function but a front-line business activity that directly shapes the in-store experience and the brand's reputation. Absorbing peak traffic from a vast store network while keeping its multi-tenant SaaS stable through frequent release cycles is the challenge the company has to meet.

Zhejiang Lemeng Information Technology Co., Ltd. ("Lemeng") is a software and information technology service provider focused on digitalization for the food retail industry. Under the Lemeng brand, the company delivers integrated retail digital-intelligence solutions for supermarkets, chain stores and trade distributors, with a product matrix that covers the full range of store operations: a retail POS suite in single-store, chain, supermarket and overseas editions, chain-store ERP, trade distribution and inventory management, a warehouse management system (WMS), a fund and accounting system (aggregated payment and split settlement), omni-channel membership CRM, and a WeChat mini-program new-retail mall, plus extended capabilities such as fresh-produce delivery, visual merchandising and AI smart scales. Its industry solutions cover twelve verticals — supermarkets, snacks, fruit, fresh produce, cooked food, roasted nuts, bakery, tea drinks, catering, meat, alcoholic beverages and trade distribution. To date the company serves more than 200,000 stores in more than 200 cities nationwide, has been chosen by more than 15,000 retail brands and more than 100 leading regional enterprises, and provides 24/7 human support.

Business Challenges

Lemeng's business model makes its O&M far more complex than that of a typical SaaS provider. Huge numbers of store terminals generate checkout and payment requests all at once during business peaks; the product line spans checkout, ERP, WMS, payment and split settlement, mini-programs and more; and observability data sits in different technology stacks. As business lines and store numbers keep growing, that complexity has climbed to a new order of magnitude.

Fragmented alert governance, with real anomalies buried in storms

Checkout SaaS, chain ERP, WMS and payment accounting each relied on a different observability system — Simple Log Service (SLS), Prometheus metrics, application performance management (APM) and cloud service monitoring — so alert rules had to be configured and maintained in separate consoles. As the business expanded, rule counts ballooned. Misconfigurations and omissions were easy to introduce and hard to detect, and traffic peaks during sales promotions and holidays set off alert storms in which engineers on duty struggled to pick out the issues that genuinely needed action.

Observability data hard to produce, hard for the business to consume

Reporting needs across internal R&D, customer success and regional operations teams vary widely, yet all of them matter: API success rates, latency distribution and resource load on the system side; store online rate, checkout transaction volume, payment success rate and split-settlement timeliness on the business side. Building such dashboards usually meant writing complex queries across several kinds of observability data — slow and error-prone. For business users, query syntax was another barrier. Observability stayed in the hands of a few experts, and response times could not keep pace with the business.

Manual monitoring before failures, manual digging after them

On the prevention side, engineers logged in and checked the container clusters and cloud service resources behind the multi-tenant business item by item. There was no reusable inspection SOP and no systematic way to spot resource bottlenecks and latent risks before a failure. On the diagnosis side, a single failed checkout in a store could originate in the store network, the POS terminal, the SaaS application, the database or the payment channel. Cross-system troubleshooting meant switching between platforms again and again and correlating data by hand, and the process leaned heavily on the experience of senior engineers. Every additional minute spent diagnosing the payment path directly affects store operations and financial compliance.

What Lemeng needed was not another monitoring tool, but an artificial intelligence for IT operations (AIOps) system that would unify the alert entry point, lower the barrier to consuming data, move the line of defense forward, and let AI handle cross-system diagnosis.

Solution: Four Layers of AIOps on a Unified Data Foundation

After evaluation, Lemeng chose Alibaba Cloud's Cloud Monitor 2.0 (CMS) together with STAROps, Alibaba Cloud's full-domain AIOps platform, as the core of this upgrade. Lemeng had already accumulated logs, metrics, traces and other observability data on the Alibaba Cloud observability stack, and STAROps uses the UModel observability model to unify them into an end-to-end digital twin that alert governance, automated inspection, root cause analysis and other capabilities then consume. The platform needs no separate activation — it is enabled by default through the STAROps entry point in the CMS 2.0 or SLS console — so it slots smoothly into Lemeng's existing observability setup. The upgrade delivered four layers of capability to address the three challenges.

(1) Unified Alert Governance: Consolidating Alerts from Multiple Systems into a Single Entry Point

With the Alert Management Center in CMS 2.0, Lemeng brought alert rules that had been scattered across SLS, Prometheus metrics, application monitoring and cloud service monitoring into one platform, so engineers no longer jump between systems to configure them separately. Lemeng also brought in the alert rule governance Skill built into STAROps: it inspects existing alert rules automatically and issues governance recommendations, flagging duplicate rules, rules with unsound thresholds and stale rules that have not fired in a long time. Combined with alert and event analysis, it consolidates alerts during peak periods, significantly reducing the drain of alert storms on on-duty efficiency.

(2) Natural-Language Observability: Generating Dashboards with a Single Sentence

Using the natural-language-to-SQL (NL2SQL) capability in STAROps, Lemeng's developers and operations staff describe what they want to observe in plain language, and the platform generates the matching queries and dashboards. Typical uses include checking the trend in checkout API success rates for stores in a given region; comparing latency distribution across payment channels during sales promotions; tallying delay distribution and failure details for split-settlement jobs; and summarizing alert handling and acknowledgment status by line of business. Producing a dashboard has collapsed from a multi-step process — file a request, wait for scheduling, write the query — into a single conversation, so observability data arrives quickly and business roles can use it directly.

(3) Scheduled Intelligent Inspection: Delegating Repetitive Checks to Digital Employees

Through the STAROps long-running task (Mission) mechanism, Lemeng lets digital employees (Agents) run inspections, analyses and change operations on their own, either on a schedule or event-driven, turning repetitive checks that once depended on people into reliable automated flows. The inspection scenarios in production include container cluster health inspection, application health inspection, cloud service resource health inspection, and quality checks on the payment and split-settlement data pipelines. Inspection results are captured as structured reports and can be compared against historical results, so resource bottlenecks and configuration drift surface before they turn into failures. For high-risk actions such as scaling and configuration changes, the human-in-the-loop (HIL) mechanism built into Mission prompts for manual confirmation, raising the level of automation while holding the safety line on changes. This layer upgrades the line of defense from manual item-by-item checking to automated protection.

(4) Dedicated O&M Digital Employee: Rapidly Narrowing Root Causes Across Systems

By combining default rules, Skills and Model Context Protocol (MCP) tools, Lemeng customized the general-purpose digital employee into a dedicated SRE agent fitted to its own business, configuring that agent's responsibilities, permissions and callable tools along its own system boundaries. When an alert fires on the checkout or payment path, the digital employee can trigger rapid root cause analysis from the alert context or from an engineer's natural-language question: using the O&M digital twin built with UModel, it correlates applications, services, resources, topology, alerts and changes in one view, and with AI analysis operators such as metric anomaly detection, log clustering, trace analysis and change retracing, it performs causal inference across systems. Troubleshooting that once relied on senior engineers' experience becomes reusable automated analysis, which directly addresses the problem of long fault-localization chains.

Value Delivered: Three Upgrades from "Reactive Firefighting" to "Proactive Autonomy"

(1) Alert Governance: From Fragmented Manual Work to Unified Standard Actions

Before, each of the four observability systems had its own alert entry point. Rule configuration depended on individual experience, misconfigurations and omissions went unnoticed, alerts flooded the screen at peak times, and engineers on duty had to sift real anomalies out of the noise.

Now, a unified Alert Management Center plus the rule governance Skill turns adding, reviewing and optimizing rules from manual work driven by individual experience into a standard, tool-supported process. Peak-time alerts are consolidated as needed, and alert storms no longer disrupt on-duty judgment. Alerts have been reduced by 70%.

(2) Data Access: From Request-and-Schedule to a Dashboard in One Conversation

Before, a business observability dashboard went through filing a request, waiting for scheduling and writing complex queries — slow to build and error-prone — and business users who did not know the query syntax could only wait for an expert to get back to them.

Now, with NL2SQL and conversational data retrieval, developers, customer success staff and operations teams get core application observability metrics and store-operations data in natural language, without knowing any query syntax. Observability has spread from a handful of experts to a much wider set of business roles, and response times are now measured in minutes.

(3) O&M Posture: From Reactive Response to Proactive Protection, with AI-Assisted Troubleshooting

Before, resource inspection meant checking items by hand, with no guarantee of coverage or consistency, and fault localization meant switching between platforms and correlating data manually, so mean time to repair (MTTR) was hard to bring down consistently.

Now, with a release every two weeks, scheduled Mission inspections keep producing structured reports and comparisons against history, configuration drift is caught early, and automated inspection provides a steady safety net for release quality. When a failure occurs, the dedicated digital employee triggers rapid root cause analysis straight from an alert or a question, and uses the digital twin topology together with AI analysis operators to narrow the search across systems — cutting the time engineers spend switching between consoles and correlating data by hand. MTTR on the checkout and payment paths has been steadily reduced to a matter of minutes.

Future Outlook: Pushing the Boundary of "Proactive Autonomy" Further

Unified alert governance, natural-language observability, automated inspection and intelligent root cause analysis are now running on the core paths. Next, Lemeng and Alibaba Cloud will deepen the work in three directions.

(1) From "Core-Path Pilot" to "Full Business-Line Coverage"

Lemeng will widen the digital employee's remit from the checkout and payment paths to more systems — chain ERP, WMS, mini-program new retail and others — and reuse the proven alert governance, inspection and root cause analysis capabilities as the standard configuration for every product line, so that AIOps becomes standard practice across all business lines.

(2) From "Technical Metric Observability" to "Business Continuity Measurement"

Building on NL2SQL cross-data-source querying, Lemeng will connect technical metrics such as API success rate and latency with business metrics such as store online rate, checkout transaction volume and split-settlement timeliness, building a measurement framework centered on business continuity so that the value of O&M can be expressed directly in business language.

(3) From "Human-Machine Collaboration" to "Higher-Level Autonomous O&M"

Within the safety frame of the human-in-the-loop mechanism, Lemeng will gradually widen the range of scenarios the digital employee closes out on its own, and keep converting the experience gathered in inspection, diagnosis and change into reusable skill assets — so that O&M know-how evolves from individual capability into an organizational asset.

From Reading Bills to Managing Cloud Spend: How STAROps Closes the FinOps Loop

ObservabilityGuy — Mon, 12 Oct 2026 08:23:26 +0000

This article introduces how STAROps enables continuous, AI-driven cloud cost governance and optimization across the FinOps lifecycle.

Once a company is on the cloud, the bill is usually visible enough. What's missing is being able to read it, chase it down, and act on it in time.

Most teams run into the same set of problems:

Finance spots a cost increase only at month-end, and platform and business teams then work backward to find the cause.
Billing fields are complex, and time windows, cost definitions, and usage units are easy to misalign.
Cost shows up only as a total per product, with no way to drill down to billing items, resources, and usage changes.
High or rising spend does not automatically mean waste, and the bill alone cannot tell you whether resource configuration matches actual load.
Inspections rely on manual triggers, so anomaly detection, root-cause analysis, and follow-up lack any continuous mechanism.

FinOps is not about making the bill look better. It is about making cost changes visible in time, explaining them clearly, and routing them back to the right team.

That also means cost governance cannot stop at bill analysis. Even after you pinpoint the product, billing item, and resource behind a cost increase, you have not yet found the optimization opportunity. You still need to look at CPU, memory, disk, connection, bandwidth, and traffic utilization to judge whether resource configuration matches actual load — and then turn that into actionable recommendations under constraints such as business purpose, capacity requirements, and service level agreement (SLA).

STAROps makes this process easy to start and sustain. Users simply ask a question, and the system handles data checks, definition validation, controlled queries, cost drill-down, resource utilization analysis, and optimization recommendations. If the data is not ready yet, it guides users through syncing the bill first, then picks up the analysis where it left off.

From "Reading the Bill" to Managing Cloud Spend

Put these problems into daily work and the picture is familiar.

At month-end the bill gets exported, finance spots a cost spike, the platform team helps break out the details, and engineering confirms whether a business change is behind it. Along the way, everyone keeps re-aligning time windows, product definitions, tag conventions, resource ownership, and cost baselines.

What STAROps adds is the full chain after "seeing the number": align cost and usage definitions, identify abnormal changes, and drill down to specific billing items and resources. Then bring in resource utilization to judge whether configuration matches load, and finally name what to check first, which direction to optimize, and what still needs confirmation. The results can also feed a recurring inspection that keeps tracking cost, usage, utilization, and optimization progress. In STAROps, a cost analysis looks more like this:

From cost changes to resource utilization and optimization recommendations, STAROps extends a single analysis into a cost-governance action you can keep tracking.

Users still ask in natural language, for example:

"Which business line's cloud cost increased most noticeably this month?"
"Simple Log Service (SLS) cost increased in the last 7 days — is it a usage problem or a change in billing item structure?"
"Which resources have no cost-allocation tag, and how does that affect cost attribution?"
"Which billing items had the usage changes that drove the cost increase?"
"Which high-cost resources have stayed at low utilization for a long time, and what optimization actions should be evaluated first?"
"Inspect cost anomalies, cost allocation dimensions, and usage changes for me every day."

What the system really has to answer goes beyond "how much." The bigger questions are: who is using these resources, why cost changed, whether resource configuration matches actual load, which items deserve a closer look, and what optimization direction to take next.

This chain runs the whole way from cost visibility to continuous governance, and it maps onto the three core FinOps phases.

Aligning with the Three FinOps Phases: Inform, Optimize, Operate

Mapped to the FinOps Foundation's three phases — inform, optimize, operate —, STAROps covers the full journey from data reliability and cost visibility to anomaly detection, optimization recommendations, and recurring governance.

FinOps phase	What the enterprise needs to solve	STAROps cost governance capabilities
Inform: see clearly	Are cost, usage, ownership, and billing cycle definitions timely and accurate?	Data ingestion checks, natural language queries, cost attribution, cost-allocation inspection, multi-dimensional reports
Optimize: find opportunities	Which anomalies, waste, and structural changes are worth acting on?	Anomaly filtering, Top N ranking, usage/cost breakdown, resource drill-down, optimization recommendations
Operate: keep it running	How to turn cost governance into a continuous mechanism	Scheduled inspections, parallel tasks, billing data preparation, task recovery, report archiving

These three phases are not a one-time exercise. The cause of today's cost increase can become tomorrow's inspection focus; a cost-allocation gap found this time feeds into the next governance cycle. Each phase reinforces the others in a loop — which is exactly what the FinOps Foundation means by a continuous process.

Inform: Make Cost Data Reliable and Ownership Clear

Reliable Data, Trustworthy Definitions

The first step in FinOps is cost visibility. But "visible" does not mean staring at a single summary table. What actually matters is knowing what data that table covers, how the numbers are calculated, and whether you can break them out by business dimension.

For an enterprise, the more important question is whether these points can be verified:

Can cost be attributed by product, project, department, account, region, and tag?
Can usage and cost be traced to specific resources and billing items?
Can trends, anomalies, and ownership be compared under one consistent set of definitions?
Can business teams understand the cost they own without having to understand complex billing fields?

In the STAROps cost-analysis pipeline, a natural-language question does not turn straight into an ad-hoc SQL query. The system first generates a structured analysis plan — specifying the time window, comparison baseline, metric definitions, aggregation dimensions, filter conditions, sorting rules, and Top N scope — and only then produces a controlled SQL query or request.

To keep results reliable, the system runs two kinds of validation before execution:

Validation type	Purpose
Semantic validation	Checks whether the time window, cost definition, usage units, currency, and dimension combination fit cost analysis semantics
Structural validation	Checks whether query fields, filter conditions, aggregation methods, sorting, and Top N stay within the controlled scope

The upside: users never have to understand the underlying billing fields or write SQL themselves, yet the system still enforces guardrails on query fields, aggregation methods, and cost definitions.

Cost Attribution

User question: Analyze this month's cloud cost by business project and compare cost distribution and changes across projects.

Agent answer (sample data): The e-commerce platform spent ¥100,000 this month, 50% of the total, up ¥20,000 from last month, with the increase coming mainly from Elastic Compute Service (ECS) and ApsaraDB RDS. The data platform spent ¥60,000, or 30%, roughly flat month over month. Internal R&D spent ¥40,000, or 20%, down ¥10,000. Recommendation: have the e-commerce platform team check the newly added compute and database resources first.

FinOps puts a lot of weight on "whoever uses it, owns it." But to make that principle real, cost first has to land on the right dimension.

For cost attribution, STAROps aggregates and filters mainly on fields already in the bill — Tag, ResourceGroup, CostUnit, OwnerID — and lays the cost distribution out for the user:

Tag-based cost allocation: aggregate by the tag field in the bill to view Top N tags, tags with abnormal increases, and the share of cost carried by values like (no tag) or none.
Resource-group cost allocation: aggregate by ResourceGroup to see which resource groups most cost falls into, and how much of it sits in the default resource group.
Cost units: queries and aggregation by CostUnit are supported at the underlying layer, so you can see how cost is distributed across cost units and how much remains unallocated.
Multi-account ownership: split cost by OwnerID across the primary account and financial sub-accounts to establish cost ownership in multi-account setups.

One clarification: without a corporate tagging standard or mapping table, the system will not assert outright that "the cost center is wrong". More precisely, STAROps first aggregates the ownership fields already present in the bill so users can see cost distribution across tag, resource group, cost unit, and account. Whether a given case is a cost allocation gap, a naming inconsistency, or an organizational mapping issue has to be confirmed against the company's own rules.

Looking at this only at month-end leaves you reacting. Put it into a scheduled inspection, and finance spends less time re-confirming numbers at close, while engineering sees the cost impact of new resources and architecture changes much earlier.

Optimize: Find the Opportunities Worth Acting On

Anomaly Detection and Multi-Dimensional Drill-Down

User question: Which ECS cost increases in the last 7 days deserve the most attention? Please drill down to specific instances and billing items.

Agent answer (sample data): ECS is up ¥10,000 (+10%) versus the previous 7 days. On instance i-example03, the "system disk" billing item rose from ¥1,000 to ¥5,000, an increase of ¥4,000 and 40% of the abnormal ECS increase. On instance i-example04, the "cloud server configuration" billing item rose ¥3,000, or 30% of the abnormal increase. Recommendation: check resource changes on these two instances and their billing items first.

A cost increase is not necessarily waste. Business activity, traffic growth, and architecture migration can all drive legitimate growth. What FinOps really needs to watch are abnormal changes that lack an explanation, follow-up, or owner.

STAROps compares cost and usage between the current period and a baseline period, analyzing change amount, change ratio, usage trends, deductions and discounts, resource concentration, and the breakdown across tags, regions, products, and billing items to flag the anomalies worth attention. It also uses the baseline amount to filter out "small-amount, big-percentage" noise — a resource jumping from ¥1 to ¥10 is a steep rise, but usually matters less than a core product line going from ¥100,000 to ¥120,000.

Once anomalies are identified, the system selects the objects that matter by amount and contribution, then drills down level by level:

Drill-down level	Question to answer
Total bill level	Has overall cost deviated from the historical baseline?
Product level	Which cloud products drove most of the change?
Billing item level	Does the change come from storage, traffic, requests, compute, processing, or deductions?
Resource level	Which resources or instances contributed most of the increase?
Cause level	Is the cause usage growth, a unit price change, reduced deductions, or new resources?

STAROps retains the cost change, anomaly contribution, and relative share at every level, so dozens of pages of line items converge into a handful of actionable checks. When SLS cost rises, for example, the analysis breaks it down further into billing items — storage, index traffic, read/write traffic, and data transformation — and cross-references them with specific projects, LogStores, resources, and tags to determine whether the main driver is usage growth or a shift in cost structure. Cost data becomes truly actionable only once you can pin it down to a specific billing item, resource, and cause.

Resource Utilization Analysis and Optimization Recommendations

User question: Analyze ECS cost for the last 7 days and find the low-utilization instances worth checking first.

Agent answer (sample data): Instance i-example01 cost ¥1,000 over 7 days, with CPU averaging 10% and peaking at 20% and memory averaging 10% and peaking at 20% — a low-utilization instance to check first. Instance i-example02 averages 40% CPU with an 80% peak and 60% memory on average, so it is not a low-utilization candidate for now. Optimization advice: first confirm i-example01's business purpose, peak cycle, SLA, and capacity requirements. If the low load is confirmed to be long-term, evaluate downsizing the instance. If it belongs to a test or staging environment that does not run continuously, evaluate scheduled start/stop as well. Before implementing, confirm data protection, dependencies, and the change window.

STAROps analyzes cost change, resource utilization, and business constraints together, taking users from "a cost anomaly was found" to "here is what to check and how to optimize".

The analysis covers:

Cost insight: pinpoint the main cost changes and the products, billing items, and resources behind them.
Utilization analysis: review trends for CPU, memory, disk, connections, bandwidth, and traffic utilization.

Users can read the results from several angles:

Identify the main cost changes and the resources that need attention.
Review resource utilization trends and load behavior.
Get actionable checks and optimization directions that fit the business scenario.
See what still needs business confirmation, so follow-up can be coordinated.

The results explain cost changes, resource utilization, which items to watch, and what still needs business confirmation.

If no Intelligent Utilization Analysis results are available, the report states the information gap explicitly and explains how to enable Intelligent Utilization Analysis.

If additional authorization is required, the report gives clear instructions so the user can complete the analysis.

Operate: Turn FinOps into a Mechanism That Keeps Running

Recurring Inspections and Long-Running Missions

User question: Every day at 9:00 AM, check the previous day's cost anomalies, cost allocation risks, and usage changes, and generate a report.

Agent answer (sample data): Today's inspection is complete: yesterday's total cost was ¥30,000, +10% versus the previous day; 2 key items were found — system disk cost +20%, and cost without an environment tag +10%. The report lists the main contributing resources and the owners to be confirmed. The next inspection will run tomorrow at 9:00 AM.

FinOps cannot rely on a month-end review. Effective cost governance has to keep running day to day.

For recurring cost inspections, STAROps uses its long-running task capability, Mission, to run multiple inspection tracks in parallel:

Inspection area	What it finds
Cost anomalies	Whether total, product, and resource cost fluctuate abnormally
Cost allocation risks	Whether tag, project, department, and account ownership is missing or abnormal
Deduction changes	Compares changes in deducted usage under resource plans, billed usage, and total usage
Structural changes	Identifies new and disappearing products, plus shifts in cost distribution across dimensions such as region and resource group

These tasks run in parallel and then roll up into a single inspection report. The report retains the analysis billing period, baseline period, current amount, baseline amount, change ratio, thresholds triggered, key dimensions, and suggested drill-down directions.

Mission suits exactly this kind of long-running, staged, resumable work: start with the total bill check, then branch out in parallel across product, resource, cost allocation, and usage; once something significant surfaces, keep drilling down.

If an inspection hits temporarily unavailable data, a Cost Manager sync refresh, or an interrupted drill-down, STAROps records the task state and the branches already completed, then resumes once conditions allow. For a production environment, that is far closer to a genuinely long-running system than "fail and rerun by hand."

Preparing Billing Data

Before any cost analysis can run, the billing data has to be queryable.

When a user starts a cost analysis or inspection, STAROps first confirms that Cost Manager has finished syncing the bill. Cost Manager syncs cloud bills into SLS, and every natural language query, structured SQL query, and recurring inspection afterward runs on that billing data.

If the billing data is not ready, the system prompts the user in the STAROps chat to enable Cost Manager, sync billing data, or update the required fields. Once the data is ready, the original analysis continues and the recurring inspection can be triggered again. What the user gets is not "this isn't available right now" but a path forward.

What STAROps Solves for Users

STAROps is not just another billing report. Think of it as a cost-analysis entry point you can question, drill into, and run continuous inspections from.

Common user problem	What STAROps changes
No idea why cost went up	Drills from the total bill down to products, billing items, resources, and usage changes
No appetite for writing SQL or learning complex billing fields	Starts the analysis in natural language; the system generates controlled queries
Worried that analysis definitions are inconsistent	Validates the time window, metric definitions, dimensions, and SQL structure
Too much detail, key items buried	Surfaces the key items using thresholds, Top N ranking, and drill-down results
Cost governance depends on someone remembering to check	Uses Mission to carry recurring inspections and task recovery
Billing data not ready yet	Guides you through enabling Cost Manager and syncing the bill in the STAROps chat, then returns to the analysis

The bottom line: cost questions no longer wait for month-end. You can ask any time, let the system query automatically, and keep inspections running. Cost spikes surface earlier, root causes get explained faster, and governance actions are captured so nothing falls through the cracks.

As enterprise cloud architectures grow increasingly complex, cost governance will shift from after-the-fact analysis to real-time awareness. That is exactly the path STAROps is building — from visible data to explainable anomalies to governance that never stops running.

From Business Alerts to Database Root Causes: How STAROps and ApsaraDB Agent Sharpen Cross-Layer Diagnostics

ObservabilityGuy — Sat, 10 Oct 2026 07:19:30 +0000

This article introduces how STAROps and ApsaraDB Agent enable precise, cross-layer root cause analysis from app alerts to database commands.

The alert fires. The frontend P95 latency spikes to 1,800 ms. You trace the call chain downstream—Redis connections are fluctuating, ApsaraDB RDS (RDS) active sessions are climbing—and the clues gradually converge on the database layer. At this point, opening more monitoring dashboards won't help; what you need is something that can run deep diagnostics against the database itself.

The problem isn't a lack of monitoring data—Simple Log Service (SLS), Application Real-Time Monitoring Service (ARMS), Grafana dashboards—it's all there. What's truly missing is something that can stitch cross-layer data into a causal chain. STAROps excels at full-stack correlation analysis, narrowing the troubleshooting scope from many components down to a few suspects. But once the evidence points to the database layer, you need a database diagnostic partner that can drill down on real metrics. Alibaba Cloud ApsaraDB Agent (product ID: alibabacloud-yaochi-agent) fills exactly that role. In this post, we first explain why cross-layer root cause analysis is hard, then walk through three real-world cases—one RDS and two Redis—escalating from single-instance diagnostics (L3) to cross-layer root cause analysis (L4).

Where Do the Difficulties of Cross-layer Root Cause Analysis Lie?

STAROps integrates metrics, logs, and traces across applications, middleware, and infrastructure for full-stack correlation analysis. That's its strong suit. But once the scope narrows to the database layer, the nature of the problem changes. "Redis anomaly" isn't enough—you need to know whether a Lua script blocked the single thread or a hot key maxed out the CPU. "RDS session spike" isn't enough either—you need to know which table is missing which index and which SQL query is doing a full table scan. These fall squarely in the specialized domain of database diagnostics, beyond what a general observability platform covers. ApsaraDB Agent supports RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB, enabling instance-level diagnostics and root cause analysis based on real metrics. Mount it as a skill on a STAROps digital worker, and a single evidence chain can reach from the application layer all the way down to a specific database command.

A digital worker is an AI operations assistant in STAROps that can be equipped with skills. Attach ApsaraDB Agent to a digital worker and it gains professional diagnostic capabilities at the database layer. ApsaraDB Agent can do far more than fault diagnosis—beyond instance diagnostics, it also covers everyday scenarios such as instance specification selection, kernel parameter change evaluation, cost analysis, schema modeling analysis, kernel parameter explanation, and backup status queries.

Frontend Service Chain and Alert Propagation Path

The cross-layer scenarios that follow all involve the same e-commerce application. Below is the frontend service call chain extracted from Cloud Monitor 2.0, which also represents the actual alert propagation path:

The frontend acts as the entry point for user requests, distributing them via frontend-proxy to business services such as wishlist and promotion. These services in turn call data stores like Redis (Tair, collectively referred to as Redis below) and RDS. When Redis experiences single-thread blocking, the downstream business services are the first to time out. Errors propagate upward layer by layer along the wishlist/promotion → frontend-proxy → frontend path, ultimately surfacing as a slow-response alert on the frontend.

The agent replies shown in the following scenarios are the final diagnostic conclusions. During actual use, the agent first displays analysis progress (querying metrics, analyzing slow commands…). Single-instance diagnostics typically complete within a few minutes, while cross-layer root cause analysis takes roughly 5–10 minutes. If the problem description is incomplete, the agent may ask follow-up questions.

Scenario 1: Abnormal Spike in Active Sessions Due to a Missing Index

Question

Analyze the cause of the abnormal spike in active sessions for RDS instance rm-j6c3l32c730ti37ur during the period 2026-07-18T12:21:30+08:00 → 2026-07-18T12:38:30+08:00

Agent Reply

The digital worker identifies that the issue clearly points to the RDS database layer and invokes ApsaraDB Agent for instance diagnostics.

Within the specified time window, active sessions on the RDS instance rose sharply from a normal level to a peak, lasting about 17 minutes before dropping back. Analysis reveals that during the alert period, a large volume of queries concentrated on the inventory table, all containing correlated subqueries on the product_id column. The client source was IP 10.0.20.210, with 540 concurrent connections executing these queries simultaneously. Because the inventory table lacked an index on product_id, every query performed a full table scan, leading to a backlog of active sessions under high concurrency. Root cause: the inventory table is missing the product_id index, and 540 concurrent high-cost correlated subqueries overwhelmed the RDS instance. Recommendation: add an index on product_id to the inventory table.

Summary

This is an L3 diagnosis—the alert directly points to a single RDS instance with no cross-layer propagation involved. ApsaraDB Agent's diagnostic path: confirm metric anomaly → analyze SQL distribution → trace the client source (IP 10.0.20.210, 540 concurrent connections) → locate the missing index (inventory.product_id). From "elevated active sessions" to "which table is missing which index," the diagnostic precision is enough to directly guide remediation. You can apply the index change right from the RDS console.

Scenario 2: Alert in Frontend, Root Cause in Redis Lua Script

Question

A slow frontend interface response alert occurred from 2026-08-10 10:31 to 10:50. Help me diagnose the root cause of the issue and provide a solution

Agent Reply

The digital worker first kicks off STAROps full-stack correlation analysis, gradually narrowing the troubleshooting scope from the frontend layer to the Redis layer through timeline comparison, topology tracing, and change-record cross-referencing. It then invokes ApsaraDB Agent for in-depth diagnostics at the Redis layer.

The in-depth Redis diagnostics reveal the following key metrics: CPU maxed out at 98 % yet throughput dropped (QPS fell from 1,200 to 180), indicating that an operation was blocking Redis's processing thread. Slow-command log analysis pinpointed the source: a Lua script with heavy computational logic monopolized the Redis single thread, and it fired twice during the alert window. Cascading propagation path: Redis single thread frozen → promotion and wishlist services hit SocketTimeout → frontend-proxy returns timeout errors → frontend response slows down. Root cause: the single thread of the managed Redis instance was monopolized by a Lua slow command with heavy computational logic, firing twice. Recommendation: refactor the Lua script to avoid heavy computation in the script body; consider splitting into pipeline batches.

Summary

This is an L4 diagnosis—the alert fires at the frontend layer, but the root cause lies in the Redis layer, involving cross-layer propagation. The diagnostic path: frontend alert → STAROps full-stack correlation analysis → scope narrowed to Redis → ApsaraDB Agent Redis metric analysis → slow-command log pinpoints Lua script blockage → cascading propagation confirmed (Redis → microservices → frontend). The diagnostic precision goes down to the command type and behavioral characteristics, rather than merely answering "Redis has a performance issue."

Scenario 3: Frontend Response Time Spike, Root Cause in Redis EVAL Command

Question

Analyze the cause of the frontend response time spike alert during the period from 2026-08-10 06:19 to 06:25

Agent Reply

Following the same troubleshooting path as Scenario 2: STAROps full-stack correlation analysis narrows the scope to the Redis layer, and ApsaraDB Agent takes over for in-depth Redis diagnostics.

Redis instance metrics show near-CPU saturation and a sudden QPS drop, a pattern similar to Scenario 2. Slow-command log analysis pinpointed the blockage source: EVAL commands. These commands contain extensive inline logic, executing complex calculations directly within Redis. The single thread is fully occupied during execution, forcing all other commands to queue. Root cause: the Redis single thread is monopolized by slow EVAL commands, causing all commands to queue. Recommendation: locate and block the client source issuing the abnormal EVAL calls, and optimize the script logic to avoid large-scale loops.

Summary

This is also an L4 cross-layer root cause analysis (RCA), with the same "alert in frontend, root cause in Redis" propagation chain and nearly identical metric behavior. Yet the underlying details differ: Scenario 2 involves heavy computation in the business's resident Lua script—fix it by refactoring the script. Scenario 3 involves a temporary inline EVAL script from an anomalous source (EVAL is Redis's command for executing inline Lua scripts)—fix it by blocking the anomalous source and moving the computation out of Redis. Same symptoms, different remediation paths.

Scenario Comparison: Same Symptoms, Different Causes

Looking at Scenario 2 and Scenario 3 together:

	Scenario 2 (10:31 Alert)	Scenario 3 (06:19 Alert)
Symptom	Slow frontend response	Frontend response time spike
Propagation chain	Redis → wishlist/promotion → frontend-proxy → frontend	Same as left
Metric behavior	CPU 98%, QPS 1200→180	CPU saturated, sudden QPS drop
Root cause	Heavy computation in resident business Lua script, monopolizing single thread (occurred twice)	Temporary inline EVAL script monopolizing single thread
Remediation direction	Refactor Lua script	Block anomalous source, move computational logic out of Redis

Same symptom, same middleware, and the root cause mechanism in both cases is "slow commands monopolizing the single thread"—yet the specific causes and remediation actions differ. This is precisely the value of command-level diagnostic precision: it pins the conclusion down to "which command, where it came from, and how to handle it," rather than merely answering "Redis has a performance issue."

Scope

ApsaraDB Agent currently supports the following Alibaba Cloud database products: RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB. Self-managed databases and non-Alibaba Cloud databases are not supported.

When the evidence is insufficient for a definitive conclusion, the agent marks the confidence level and lists potential troubleshooting directions rather than forcing a root cause. Before executing any fix, it is best to have a DBA or engineering lead review the diagnostic conclusions.

Quick Start

Log on to the STAROps console and create a digital employee.
Search for alibabacloud-yaochi-agent under "Skills → Features" and associate it with the employee.
Ask your question in the dialog box. Single-instance diagnostics typically return results within a few minutes; cross-layer root cause analysis takes a bit longer (roughly 5–10 minutes for the scenarios in this post).

You can use the following template to construct your prompt, replacing the content in the brackets:

Analyze the cause of [Anomaly Description] for [Instance Type: RDS/Redis/Tair/MongoDB/PolarDB] instance [Instance ID] during the period from [Start Time] to [End Time]

For example: "Analyze the cause of the abnormal spike in CPU utilization for the Redis instance r-xxxxx during the period from 2026-08-10 10:00 to 11:00." Describe the problem you encountered and the time it occurred using natural language. If you are unsure of the specific instance ID, you can describe the application name and the anomalous symptoms, and the Agent will attempt to correlate it with the underlying database instance.

ApsaraDB Agent raises database-layer diagnostic precision from "some middleware has a problem" to "which specific command or index caused the issue." STAROps, in turn, stitches cross-layer evidence into a complete causal chain. Together, SREs and DBAs can use the same toolset for end-to-end troubleshooting—from alert to fix. For complex faults involving cross-layer propagation, this combination is worth a try.

STAROps currently offers new users 10,000 credits for the first month plus a recurring free tier of 2,500 credits every month. Give STAROps a spin—feed it your most recent production incident and see whether it can actually help: pinpointing the exact command, tracing where it came from, and telling you how to fix it.

Try it now: https://starops.console.aliyun.com/

Zero-Code Transformation: Making the AI Agent Sandbox No Longer a Black Box

ObservabilityGuy — Fri, 18 Sep 2026 06:06:51 +0000

This article introduces Alibaba Cloud's OBI for zero-code observability of AI Agents in ACS Agent Sandboxes.

In 2026, AI Agents are making the critical leap from "chatting" to "doing": writing and executing code themselves, operating browsers to search for information, and invoking a series of tools to complete complex tasks. The prerequisite for allowing an Agent to safely "take action" is providing a secure and efficient execution environment. Consequently, sandboxes have become the de facto standard for Agent infrastructure.

Agent Sandbox, launched by Alibaba Cloud Container Compute Service (ACS), serves as the sandbox compute foundation for production-grade AI Agents, featuring MicroVM-level isolation, elasticity of 15,000 instances per minute, and memory-level hibernation and wake-up. However, a new problem arises: the better the isolation, the more the sandbox interior resembles a black box. Operations and security teams find it difficult to see which models the Agent invoked, what code it ran, and which external services it accessed.

Now, this puzzle piece is in place: OpenTelemetry eBPF Instrumentation (OBI) from Cloud Monitor 2.0 (CMS) officially supports the ACS Sandbox scenario. Without changing a single line of code or rebuilding images, every model invocation, tool call, and external access by an AI Agent within the sandbox automatically generates standard traces and metrics.

Sandbox Foundation: Isolation, Elasticity, and State

Code executed by AI Agents has a distinct characteristic: much of it is dynamically generated by large models on the spot. Such code is untrusted and unpredictable, and must never run in the same container as production workloads. Agent Sandbox addresses this with a MicroVM-level isolated execution environment: each sandbox has a dedicated kernel with end-to-end isolation across compute, network, and storage. Even if the code inside exhibits abnormal behavior, it remains securely locked in the "box."

Beyond security, there are three sets of capabilities tailored for Agent scenarios.

Large-scale, low-latency elasticity. Agent tasks are often bursty: a reinforcement learning session (AgentRL) might instantly require thousands of environments, while peak requests for an online service (Agent Serving) can surge. Agent Sandbox supports creating up to 15,000 sandboxes per minute. Combined with a warm pool, it achieves hundred-millisecond startup times, and image caching reduces pull times by over 90%. In a public case study, MiniMax spun up 5,000 sandboxes within 10 seconds.

State retention. Running sandboxes support on-demand hibernation with memory state fully preserved, enabling rapid wake-up within 1 to 10 seconds. Combined with checkpoint/restore, an Agent's execution state can be saved, migrated, or cloned. When multi-turn conversations are interrupted, there is no need to destroy and rebuild the environment. No CPU or memory fees are charged during hibernation, and per-second billing makes "use it and pause it" truly cost-effective.

Open ecosystem. Agent Sandbox provides two integration methods: an E2B-compatible SDK and the Kubernetes-native Sandbox CR. It is compatible with mainstream Agent frameworks such as AgentScope and LangGraph, covering typical sandbox scenarios like Code Interpreter (code execution) and Browser Use (browser operations).

Monitoring Blind Spots: The More Secure, the More Like a Black Box

The sandbox solves the problem of "whether it is safe to run," but "how well it runs" remains shrouded in mystery. Attempting to apply traditional Application Performance Management (APM) solutions one by one, you find that all three paths lead to dead ends.

SDK instrumentation cannot get in. Traditional application monitoring relies on integrating SDKs or attaching language-specific instrumentation agents in the code. However, the code running inside the sandbox is generated by large models on the fly — it is diverse and unpredictable. Agent frameworks themselves also span multiple languages such as Python and Node.js. You cannot expect the model to "add monitoring instrumentation while generating code."

Host-level collection has nowhere to go. ACS is a serverless container compute service with no user-visible nodes, leaving traditional DaemonSet-based host-level monitoring components with no foothold. The strong isolation of MicroVM completely blocks the path of "looking in from the host."

Sandbox lifecycles are too short. Created in hundreds of milliseconds, destroyed after use, hibernated at any time — by the time you log in to troubleshoot, the environment has long been gone. A host-centric monitoring model simply cannot keep up with this ephemeral rhythm.

The result: when an Agent task fails, you do not know whether it was a slow model response, a tool timeout, or a code error. When token costs rise, you cannot tell which stage consumed them. When the security team wants to audit which external addresses the sandbox code accessed, they can only sift through network logs manually.

The OBI Solution: Achieving AI Agent Observability with Zero Modification

OBI takes a different approach: instead of entering the application, it stands between the application and the kernel.

OBI intercepts application communications at the kernel and library function levels, parses protocol semantics, and directly outputs standard OpenTelemetry telemetry data. This means that regardless of whether the sandbox runs Go, Java, Python, Node.js, or .NET, and regardless of the HTTP framework, database, or large model provider involved, monitoring data is automatically collected without modifying any code. For sandbox scenarios where "code is dynamically generated by models on the spot," this is virtually the only viable solution.

In the ACS Sandbox environment, OBI runs alongside the sandbox workload as a sidecar. After installing the ARMS instrumentation onboarding assistant ack-onepilot (v5.2.2 or later), you only need to add a few labels to the workload YAML. The OBI sidecar is then automatically injected, and the collected data is centrally aggregated in the CMS console. (Running eBPF inside a sandbox requires specific Linux kernel permissions, which can be enabled for your account by submitting a ticket.)

Capability List: Traces, Metrics, and GenAI Semantics

What can you see after integration?

The first layer is the full suite of classic application monitoring: application topologies, traces, abnormal transactions, slow transactions, and SQL analysis. Every external call by the Agent inside the sandbox generates a trace — whether the bottleneck is model inference or tool execution, a single trace reveals it all.

The second layer is the true highlight of this upgrade: AI Agent observability. OBI features built-in protocol-level tracing for four major GenAI providers — OpenAI, Anthropic, Google Gemini, and Qwen — automatically identifying LLM calls and generating telemetry data according to GenAI semantic conventions:

LLM invocations: Automatic extraction of model names, token consumption, latency, and error states;
Tool calling: Automatic parsing of tool calling information from model responses, making the Agent's "decision on which tool to use" clear at a glance;
Model Context Protocol (MCP), embedding, reranking, and vector retrieval: full coverage of the core stages in the RAG pipeline.

If your model traffic flows through self-hosted inference services or LLM gateways such as vLLM or LiteLLM, you only need to declare the corresponding host in the configuration. OBI parses them using OpenAI-compatible protocols and includes them in the AI Agent observability view.

For sandbox scenarios, this data simultaneously answers three questions: Performance—which stage is slowing down the task; Cost—which types of invocations are consuming tokens; Security—which external addresses the untrusted code in the sandbox accessed. All of these are recorded out-of-band, naturally serving as an audit ledger.

Three-Step Integration: Install, Label, and View Data

The entire integration process requires no changes to the application code.

Step 1: Install the ack-onepilot component (v5.2.2 or later) for your ACS cluster in the ACS console.

Step 2: Add OBI labels to the YAML file of the target workload:

labels:
  apsara.apm/application-type: obi    # Indicates that this application is integrated via OBI 
  armsPilotAutoEnable: 'on' 
  armsPilotCreateAppName: "your-application-name" 
  armsPilotAppWorkspace: "your-workspace-name"

Step 3: Wait about 2 minutes, then open the CMS console: view topologies and traces under "Application Monitoring," and view model and tool call details under "AI Agent Observability."

Step 4: Go to the AI Agent application in CMS to view monitoring data:

Final Thoughts

Competition in Agent infrastructure is moving from the "make it run" phase to the "make it transparent" phase. ACS Agent Sandbox ensures that AI Agents run securely and elastically, while OBI makes every execution transparent and visible. Together, "security" and "observability" for production-grade Agents can coexist for the first time — all at the cost of zero lines of code.

If you are building Agent applications, start with the official documentation "Access ACS applications through OBI": https://www.alibabacloud.com/help/en/cms/cloudmonitor-2-0/access-acs-applications-through-opentelemetry-non-intrusive-monitoring-obi

Toward Production-Grade AgenticOps: How STAROps Builds Generalizable Root Cause Analysis

ObservabilityGuy — Wed, 16 Sep 2026 07:23:26 +0000

This article introduces STAROps, an AgenticOps framework for reliable, generalizable root cause analysis in production.

When people evaluate AgenticOps, they often start with the obvious questions: How many tools can the agent use? Can it run automated inspections, propose a fix, or even carry out a change?

But as we built STAROps, we kept coming back to a more fundamental question: Does it identify the right root cause?

We see root cause analysis (RCA) as the core of AgenticOps because it determines where every subsequent step is directed and what that step does. If the root cause is wrong, impact evaluation, remediation, change execution, and outcome validation all focus on the wrong target. When an agent can only advise, a wrong RCA wastes engineers' time. When an agent has permission to act, a wrong RCA becomes a production risk.

RCA sets the direction. Every other capability determines only how quickly and how far the agent can follow it.

When the Root Cause Is Wrong, Every Action That Follows Is Wrong

An alert usually tells us which metric crossed a threshold, not where the failure began.

A frontend endpoint returning 5xx errors may point to a problem in the frontend code, but it could also be caused by a slow downstream database, an unavailable cache, exhausted node resources, or a recent configuration change. High CPU is not necessarily the root cause either: it may be the result of a traffic spike, or it may coincide with a full garbage collection cycle, thread pool exhaustion, or a runaway loop.

Those distinctions change the response. Mistake a node resource problem for insufficient application capacity, and you may keep scaling out pods that run on the faulty node. Mistake a slow SQL query for a frontend code defect, and you may trigger a rollback that changes nothing. Mistake a traffic drop caused by an accidentally scaled-down Deployment for a network problem, and the investigation starts in the wrong direction.

General-purpose agents can already query a wide range of data and produce well-structured reports that sound plausible. But a complete report does not prove that the agent found the object that actually failed. If it stops at the entity where the alert fired, or mistakes a pronounced anomaly along the propagation path for the root cause, more elaborate reasoning can make the wrong conclusion even more convincing.

Why a Proof of Concept Cannot Prove RCA Capability

When teams validate RCA, they usually choose a handful of known faults: a Pod stuck in a crash loop, a slow SQL query, or a Redis outage. They then check whether the system returns the expected answer.

This is a reasonable way to validate a proof of concept. The scenarios are well defined and the results are easy to verify, so teams can quickly confirm that the data, tools, and product workflow operate end to end. But it proves only that the system handles scenarios A, B, and C. It does not show that the system will know where to look when fault type D appears.

If every alert type is mapped to a Skill or workflow, a proof of concept will usually perform well. The alert type, investigation entry point, query sequence, and expected answer are all specified in advance, so the agent only has to follow a path that someone has already designed. Change the alert, or let the same fault surface at a different point in the business, and that path may no longer work.

That does not mean Skills lack value. Experienced site reliability engineers (SREs) know which metrics to check first when latency rises, when a change must be examined, and which symptoms tend to send an investigation down the wrong path. That experience should be captured. Skills work best as investigation leads and checklists that help the agent avoid dead ends; they should not define the limits of RCA.

Real Production Faults Follow No Fixed Playbook

In production, however, far more things can go wrong. A problem may originate in an application, container, node, database, cache, or network. It may also follow a release, a scaling operation, or a configuration change. Topologies keep changing, data may be incomplete, and several anomalies can appear in the same time window.

The same alert can have entirely different causes, and the same root cause can trigger different alerts in different systems. The root cause may be in a database while the alert fires on the frontend. It may be on a node while the first user-visible symptom is a traffic drop on a business endpoint. Real faults routinely cross data domains and propagate through call, deployment, and host relationships.

Generalization does not mean building a model that knows the answer to every fault. A more realistic standard is that, when no workflow matches exactly, the agent still knows how to begin. It first identifies the entity that the alert refers to, then follows relationships to the relevant services and resources. When several candidates emerge, it gathers more evidence, checks the sequence of events, and eliminates hypotheses that cannot explain the symptoms. If the evidence is insufficient, it knows when to stop instead of forcing a root cause.

Fault scenarios cannot be exhaustively enumerated, but the basic method of investigating them can be reused. STAROps is designed to keep an investigation moving and converge on an answer even when the fault is unfamiliar.

How STAROps Turns RCA into a System Capability

Because a scenario catalog can never cover every production fault, simply adding more workflows cannot be the primary goal. STAROps focuses on the investigation itself: how the agent understands the current system, decides what to examine next, demonstrates that a conclusion is reliable, and learns from failures in production.

First, Understand the System: UModel

Many RCA investigations go wrong at the first step because different systems use different names for the same object.

In application performance management (APM), a business service appears as a service. In distributed traces, it appears as a set of spans. In Kubernetes, it maps to a Deployment and its pods; further down, it maps to an ECS instance or a node. Logs, metrics, and change records use different fields as well. Without a unified model of these object relationships, the agent has to match objects using names and context. It can easily split one object into several or mistake a neighboring object for the root cause.

UModel organizes these objects and relationships. It records which pods run a service, which Deployment each pod belongs to, which node hosts it, which databases the service depends on, and where to query the relevant metrics, logs, and traces.

The immediate benefit is that the agent knows exactly what it is investigating. Whether the investigation moves down from a business endpoint to a service, pod, and node, or outward from a change to the affected services, it can follow explicit relationships. The model proposes hypotheses and interprets evidence; UModel provides a stable map of the system.

Static Topology Is Not Enough: Map the Investigation, Too

A console topology usually shows which objects exist and which ones depend on others. A fault investigation needs to capture something else as well: which objects have already been checked, where anomalies were found, which candidates are under suspicion, and which branches remain unexplained.

On top of UModel, STAROps maintains a dynamic investigation topology. The investigation starts with the alerting entity and expands as evidence accumulates. When a candidate is ruled out, that branch stops. If several pods on the same node degrade at the same time, the investigation converges on the host. If database connections slow down, the agent checks the SQL queries, connection pool, and upstream and downstream traces.

This graph serves as both an investigation map and a record of the process. Metrics, logs, traces, events, and changes are attached to the relevant objects. Candidate root causes are retained along with the evidence for and against each one. Engineers can see why the agent continued in a particular direction, add information at critical points, or take over the investigation.

Use a Benchmark to Expose Answers That Only Look Right

RCA agents face a tricky problem: an answer can be wrong even when the explanation sounds convincing. From the final report alone, it is often hard to tell whether the agent followed the evidence to the root cause or guessed a common answer from the alert's surface symptoms.

That is why RCA-Bench records more than a single root cause label. The public RCA-100 dataset contains 103 fault cases across six major categories and 28 fault types. Each case records the root cause entity, fault type, propagation path, and key evidence. In a queryable environment, the agent must decide for itself what to examine first and where to go next.

We evaluate three things separately: whether the agent found the right root cause entity, identified the right fault type, and supported its investigation with evidence. The first two are scored mainly from entity relationships and fault-type relationships in UModel. Deterministic rules currently account for about 82% of the overall score. The LLM is used only to help judge the direction of the investigation and whether the evidence is sufficient, so one model does not have complete authority over another model's score.

Production Failures Must Feed the Next Iteration

An offline benchmark can show whether a capability exists, but production introduces permissions, data integrations, customer-specific topologies, and version differences. A version that scores well offline may still fail in production because it cannot retrieve the data, chooses the wrong tool, or converges too early.

STAROps retains the task input, tool calls, query results, errors, evidence, execution time, and user feedback from production runs. For a low-scoring run or failure case, we first determine whether the problem lies in object identification, data retrieval, investigation direction, evidence quality, or the final conclusion.

Reproducible failures become regression cases. Whenever UModel, a tool, a Skill, a model, or an investigation strategy changes, we validate the change against those cases. This turns a one-off fix into a regression check that every later version must pass.

Head-to-Head Results: A Complete Report Does Not Mean an Accurate Root Cause

The published RCA agent benchmark uses a stratified evaluation set of 30 cases drawn from RCA-100. STAROps and OpenClaw + DeepSeek-V4-Pro receive the same task inputs and are evaluated with the same brise scorer.

Evaluation dimension	STAROps	OpenClaw + DeepSeek-V4-Pro	Difference
Overall score	75.23	51.02	+24.2
Root cause entity	90.	52.8	+37.2
Fault type	58.	33.3	+24.7
Investigation process	75	69.8	+5.2

The totals matter, but the breakdown matters more.

The gap in the investigation-process component is only 0.052, equivalent to 5.2 points in the table. This shows that a general-purpose agent can already perform multiple rounds of queries and produce a reasonably complete analysis. The real separation comes from identifying the root cause entity and the fault type. Choose the wrong object, and every subsequent metric, log, and trace centers on that object. Misclassify the fault type, and the remediation recommendation loses its foundation.

By major fault category, STAROps leads by 50.6, 29.9, 27.6, and 24.2 points in database, node, code, and resource cases, respectively. It trails the comparison system by 11.2 points in traffic cases, mainly because of rate-limiting scenarios. Rate limiting, DNS, CDN, third-party services, and public network paths are the areas STAROps will focus on next.

Case Studies

Hard cases do not necessarily involve obscure technologies. The real difficulty is often that the alert fires far from the root cause while several plausible explanations appear at the same time. The following two cases fit that pattern.

product-catalog Traffic Drops, but the Root Cause Is an Underlying Node

The high-node-CPU case begins with a traffic-drop alert for product-catalog::ListProducts.

If you look only at the business-level data, frontend is the most obvious suspect. Its traffic and error rate change at the same time, and it is the upstream entry point for product-catalog. The general-purpose ReAct agent concluded that the load balancer had failed, while OpenClaw attributed the problem to frontend HTTP 5xx errors. Both reports cite data and describe a propagation path that sounds plausible.

The actual root cause is Kubernetes node cn-hongkong.10.0.1.107. CPU utilization on that node rose from about 10.38% to 99.98%, affecting multiple pods on the same node. frontend traffic dropped by 66.66%, followed by a 62.48% drop in ListProducts traffic, which ultimately triggered the business alert.

STAROps did not stop at frontend. It moved from the alerting operation to its service, then followed the runs-on relationship to the pod and host node. Other nodes remained healthy while several pods on this node degraded at the same time. Together, those two pieces of evidence narrowed the investigation to the infrastructure layer.

In this evaluation, ReAct scored 15, OpenClaw scored 12, and STAROps scored 94. If the team had acted on either of the first two conclusions, it might have kept investigating the load balancer or scaled out frontend while leaving the node with saturated CPU untouched.

Node CPU stays near 100% 
↓ 
Multiple pods on the same node are affected 
↓ 
frontend and other business services degrade at the same time 
↓ 
product-catalog API traffic drops, triggering an alert

Frontend Checkout Slows Down, but the Root Cause Is a Slow SQL Query

The slow-SQL case begins with an alert on frontend::POST /api/checkout. At the same time, the environment shows changes in checkout latency, Kafka lag, Java Virtual Machine (JVM) garbage collection, thread count, and Deployment scaling. Any one of these signals, viewed in isolation, could look like a plausible root cause.

The general-purpose ReAct agent concluded that checkout contained a code defect, while OpenClaw located the problem in frontend's checkout logic. Both captured part of the anomaly, but their investigations stopped midway along the propagation path.

STAROps continued downstream through the traces. checkout depends on cart, which in turn depends on inventory. A slow inventory trace showed a SELECT query taking about 10.8 seconds, plus another 2.1 seconds to acquire a database connection. The slow query held up inventory, and the delay then propagated through inventory → cart → checkout → frontend, ultimately appearing as a slow response from the frontend endpoint.

There were several distracting signals along this path. Kafka lag was measured only in milliseconds and could not explain a timeout measured in seconds. The garbage collection, thread-count changes, and Deployment scaling occurred after the services had begun to degrade, making them more likely consequences of the slow query. Only after the signals were placed on a single timeline did the evidence converge on the slow SQL query in inventory.

ReAct and OpenClaw both scored 15 on this case, while STAROps scored 84. If the team had treated the problem as a code defect, it might have rolled back frontend or checkout while leaving the SQL query that was blocking the call chain in place.

inventory slow query and blocked connection acquisition 
↓ 
cart calls to inventory slow down 
↓ 
checkout call chain latency increases 
↓ 
frontend checkout slows down, triggering an alert

Conclusion

One of these faults occurred at the node-resource layer; the other occurred along the database access path. They have little in common on the surface, and STAROps used different data and investigation paths for each. But the method was the same: identify the object, then gather evidence along its relationships. When multiple candidates emerge, place them on a single timeline and determine which one explains the full propagation path and which ones are merely accompanying symptoms.

This is what generalization must solve. We cannot make an agent know every possible fault in advance. We can, however, ensure that it can move an investigation forward when the problem is unfamiliar and stop at the right boundary when the evidence is insufficient.

That is why RCA sits at the core of STAROps. Impact analysis, remediation recommendations, and change execution have a reliable starting point only when the agent consistently identifies the right object and cause. As agents gain more permission to act in production, the cost of a wrong root cause rises. A single misjudgment can escalate from misleading an engineer to triggering the wrong scale-out, rollback, or configuration change.

We expect the capability gap in AgenticOps to center increasingly on judgment quality. Tool calling and automated execution will become baseline capabilities. What will determine whether an agent is truly ready for production is whether it can build a complete evidence chain for an unfamiliar fault, distinguish the root cause from accompanying symptoms, and stop when the evidence runs out.

For STAROps, root cause identification is an engineering discipline that must be continuously calibrated against real faults. Keeping its judgments accurate and its limits explicit remains our most important task as we move AgenticOps into production.

Plaintext Stays Put, Logs Stay Useful: Sensitive Data Protection with Alibaba Cloud Observability

ObservabilityGuy — Fri, 11 Sep 2026 05:22:45 +0000

This article explains how Alibaba Cloud SLS secures sensitive AI logs via multi-stage masking and encryption while preserving analytical value.

AI Agents are filling logs with more sensitive data in more complex forms. Beyond traditional fields such as phone numbers, IP addresses, and API keys, AI coding Agents, retrieval-augmented generation (RAG) applications, and operations-focused AI Agents may also read source code, configuration files, environment variables, and database connection strings. They then write tool calls, terminal output, and model responses to audit logs, making it easier for sensitive information to enter the logging pipeline.

You cannot simply discard this data; it provides essential evidence for troubleshooting, auditing, and governance. To keep data both secure and useful, Alibaba Cloud Observability protects sensitive data across collection, ingestion, and transformation. Data can be masked or encrypted before it reaches a LogStore, then transformed after ingestion into secure datasets suitable for long-term retention and shared analysis.

Let's start with the overall design, then look at how each method works in practice.

Reduce Exposure Without Sacrificing Value: Protect Sensitive Data End to End

The Simple Log Service (SLS) approach to sensitive data protection comes down to two questions: how should you protect the content, and where in the data pipeline should that protection happen? The first determines whether to use masking, symmetric encryption, or envelope encryption. The second determines whether to process the data on the client side, in an ingest processor, or through data transformation.

You can combine these two dimensions flexibly. The same protection method can run during collection, before ingestion, or after data reaches a LogStore; at any of these stages, you can choose masking or encryption based on the risk. Match the combination to the data source, compliance requirements, and downstream analysis. Use client-side processing when plaintext must not leave the host, an ingest processor to govern direct writes from multiple sources, and data transformation to create a protected LogStore for long-term sharing or governance of existing data.

Choose Based on How the Data Will Be Used

Different types of sensitive data call for different treatment. SLS provides three types of SPL functions: masking, symmetric encryption, and envelope encryption. These options address three needs: preserving readability, allowing authorized recovery, and separating duties.

Masking: Keep Logs Readable and Reduce Privacy Risk

Use mask for personally identifiable information (PII) or credentials scattered through free text. It supports two matching modes:

buildin: matches built-in patterns for mobile phone numbers, email addresses, national ID numbers, bank card numbers, IP addresses, landline numbers, AccessKey pairs, license plates, international mobile subscriber identity (IMSI) values, vehicle identification numbers (VINs), and more;
keyword: matches field or key names such as password, token, authorization, and api_key.

You can replace an entire value with a placeholder or retain a prefix and suffix. Keeping the first three and last four digits of a phone number, for example, protects privacy while still allowing a rough cross-check against application records during troubleshooting. Masked logs remain readable, searchable, and aggregatable, making masking a good default for most logging scenarios. For implementation details, see the page titled Use the mask function for data masking.

Common use cases include:

phone numbers, email addresses, and national ID numbers embedded in AI gateway prompts;
client IP addresses and authentication tokens in application access logs;
bank card numbers, addresses, and user identifiers in LLM application logs.

Symmetric Encryption: Store Ciphertext and Decrypt It When Authorized

Use aes_encrypt when data must be recoverable later and key ownership is clearly defined. Content such as internal RAG Q&A, model requests, and customer solution documents can be stored as ciphertext, then decrypted during an authorized audit by the team that holds the key and initialization vector (IV).

If one team handles collection, analysis, and auditing, it can manage the key and IV centrally, making symmetric encryption a simple, high-performance choice. The boundary is equally clear: once the key is shared, every holder can decrypt the data. This makes symmetric encryption suitable within one organization or security domain, but not for broad distribution across teams.

Envelope Encryption: Separate Public and Private Keys for Collaboration on Highly Sensitive Data

Use envelope_encrypt when data collection and decryption are handled by different parties. It generates a random data encryption key (DEK) to encrypt the application data, then encrypts the DEK with an asymmetric public key. The collection side holds only the public key; a separate security or audit team safeguards the private key.

This pattern is well suited to highly sensitive content, including:

the full output returned after an AI coding tool reads a file;
command output;
entire log entries containing keys, connection strings, or configuration files.

Envelope encryption also provides predictable performance. A DEK is generated at the pipeline level, and asymmetric encryption applies only to that key; AES still encrypts the application data. This minimizes the effect on collection throughput.

Protection methods are not tied to specific processing stages. The following sections trace three points along the data path and use one representative method for each, but every stage can support other methods as well.

Client-Side Protection: Keep Plaintext at the Source

In stricter compliance environments, protection must happen at the collection source. Logs are processed as soon as they are collected, so plaintext never leaves the host or container or enters the downstream transport pipeline.

Client-side protection comes in two forms. LoongCollector processes host and container logs as they are collected, while LoongSuite Pilot provides on-device privacy controls for AI coding Agents. Together, these approaches contain plaintext before data leaves its source.

LoongCollector: Process Data as You Collect It Without Changing the Application

Note: All names, phone numbers, email addresses, public keys, private keys, and other sensitive data shown in this article are synthetically generated.

For applications running on Elastic Compute Service (ECS) instances, Container Service for Kubernetes (ACK) clusters, on-premises data center hosts, or containers, LoongCollector can run SPL during collection instead of acting solely as a collector. AI gateways, Alibaba Cloud Model Studio applications, RAG services, and model proxy layers typically produce large volumes of access and interaction logs. These logs combine structured fields with natural-language input. Processing them at the collection source significantly reduces plaintext exposure.

Consider an AI gateway access log. A raw entry might contain the caller's IP address, an API key, and the user's prompt:

model=qwen-max client_ip=203.0.113.55 prompt="My name is Zhang. My phone number is 138xxxxxxxx, and my email address is zhangwei@xxxx.com. Please look up my recent orders." tokens=128

In the LoongCollector collection configuration, mask can handle built-in PII patterns and keyword fields in a single pass:

After the log is written to SLS, it can still be used to analyze model call volume, token usage, and gateway access trends, but the sensitive values have been replaced:

This approach works well for host- and container-side data such as application logs, gateway logs, and model proxy logs. It requires no changes to application code: processing happens in the collection configuration, and logs reach SLS in a safe, masked form.

LoongSuite Pilot: Mask AI Coding Data Before Distribution

Another common client-side use case is auditing AI coding activity. Coding Agents running on a developer's machine often access source code, configuration files, and local environment variables. If a developer pastes a configuration snippet into a prompt, or an Agent reads a .env file and writes its contents into tool-call arguments, API keys, cloud-provider access key/secret key pairs (AK/SK), and private keys can enter the collection pipeline.

LoongSuite Pilot is designed for these on-device AI coding scenarios. It discovers and collects activity data from Claude Code, Cursor, Codex, Qoder, and similar tools, then normalizes the data into audit events for sessions, tool calls, model calls, and token usage. Its built-in rule-based masking engine scans and replaces sensitive content before sending data to any output channel. Once masking is enabled, the same rules apply whether the destination is SLS, JSONL, HTTP, or OpenTelemetry Protocol (OTLP).

Pilot's automatic masking covers these common credential types:

Masking type	Coverage	Replacement marker
Cloud AccessKey	Alibaba Cloud LTAI, AWS AKIA/ASIA, Tencent Cloud AKID	`[ACCESSKEY_MASKED]`
API key	OpenAI-compatible `sk-` keys; GitHub personal access tokens (PATs) with `ghp_`, `gho_`, or `ghs_` prefixes	`[APIKEY_MASKED]`
Database connection string	MySQL, PostgreSQL, MongoDB, and Redis URIs; password-bearing JDBC URLs	`[DATABASEURL_MASKED]`
Private key	PEM/OpenSSH private key blocks	`[PRIVATEKEY_MASKED]`

For example, in an AI coding session, an Agent reads a local .env file and produces the following tool result:

DB_URL=mysql://root:Abc@2025!@10.0.0.12:3306/prod
ALIYUN_AK=LTAI5tFakeExampleKey0001
-----BEGIN RSA PRIVATE KEY-----
MIIEvQIBADANBgkqh...
-----END RSA PRIVATE KEY-----

After LoongSuite Pilot masks the data, SLS receives:

mask.mode controls masking, with three supported values: none, all, and custom. Masking is disabled by default. Once enabled, sensitive credentials no longer reach downstream outputs in plaintext. Audit metadata such as session_id, tool_name, token usage, and execution duration remains available, so engineering managers and security teams can still analyze AI coding activity, tool-call risks, and cost distribution.

In summary, client-side protection is a good fit when:

plaintext must not leave an application host or developer machine;
the collection pipeline crosses network and team boundaries, making plaintext transmission unacceptable;
you want to protect sensitive data at the point where logs are created without changing application code.

Protect Data Before Ingestion: Govern Direct Writes Centrally

Not all data passes through LoongCollector. Many systems write directly to SLS through SDKs, web tracking, OpenAPI, or third-party collection pipelines. For such data, an ingest processor provides centralized server-side governance: after the data reaches SLS but before it enters a LogStore, the processor runs SPL to mask or encrypt it.

An ingest processor's key advantage is centralized configuration. Rules live in SLS, so clients do not need to be upgraded one by one. When a processing policy changes, you update the processor once to cover all direct writes. This is easier to roll out across multiple business lines, SDKs in different programming languages, and third-party systems.

Consider an AI gateway access log. The gateway joins the fields for each model call into a line separated by ## and writes it directly to SLS. The request field contains the complete model request body and may include arbitrary sensitive content, such as the user's prompt or national ID number. The other fields contain structured call metadata:

2026-06-25 11:17:25.471##c9d7ab4c-b312-9d08-9243-c7f7475905be##5000000276111136##API_KEY##qwen-max##text-generation##{"input":{"prompt":"My national ID number is 53010219200508011x. Please check my insurance policy status."}}##chat-app##3de69809a9d0fd22773f33c77dab12a8

Here, request contains the entire unstructured request body. To show how an ingest processor can apply one policy to all direct writes before storage, this example uses symmetric encryption for the entire request field: it first splits the line on the delimiter, then applies aes_encrypt only to the sensitive request body.

Encrypt a Highly Sensitive Request Body in an Ingest Processor

Ingest processor SPL configuration:

* | extend temp = split(content, '##')
  | extend time          = temp[1],
           request_id    = temp[2],
           caller_uid    = temp[3],
           source        = temp[4],
           model         = temp[5],
           api_type      = temp[6],
           request       = temp[7],      
           service_name  = temp[8],
           trace_id      = temp[9]
  | extend request = to_base64(aes_encrypt(
        cast(request as binary),
        cast('aZ4kP9mX2qL7nR1t' as binary),
        cast('Gh8dW3sB6vN0cT5e' as binary)))
  | project-away content, temp

After ingestion, request is Base64-encoded ciphertext, while call metadata such as request_id, region, model, service_name, and trace_id remains in plaintext:

Operations engineers can troubleshoot the request path using request_id and trace_id, while business operations teams can count calls by model and service_name. The request body itself remains encrypted.

Decrypt Data Only When Authorized

When the original data is needed, an authorized team member who holds the key and IV decrypts it at query time:

Data Transformation: Build Secure Datasets for Shared Analysis

Client-side processing and ingest processors protect data before storage. Data transformation addresses a different problem: how to reorganize data already in a LogStore into datasets that are safer and better suited to long-term analysis.

In practice, many teams maintain two LogStores:

Raw LogStore: retains data briefly, has strict access controls, and is used only for emergency troubleshooting or security investigations;
Protected LogStore: stores the output of a data transformation task; its sensitive fields are masked or encrypted, and it retains data longer to support analysis and audits for a broader audience.

This pattern works especially well for AI observability data. Consider AI coding activity logs collected by LoongSuite Pilot from tools such as Qoder. Each llm.response event contains structured metadata, including the session ID, model, and user ID, plus a gen_ai.output.messages field. That field contains the full text of the model's response for the current turn and may include internal system names, host details, configuration data, keys, or other sensitive content:

{
  "event.name": "llm.response",
  "gen_ai.session.id": "58bf461a-285f-468e-963c-00cd1706a9d9",
  "gen_ai.provider.name": "qwen",
  "gen_ai.response.model": "auto",
  "user.id": "1",
  "gen_ai.output.messages": "[{\"role\":\"assistant\",\"parts\":[{\"type\":\"text\",\"content\":\"Candidate salary table prepared: Zhang （phone number 138xxxxxxxx）...\"}],\"finish_reason\":\"end_turn\"}]",
  "trace_id": "a58ed78ac2573159b4fa8b38ab4c2697",
  "__time__": "1782367255"
}

Because gen_ai.output.messages contains a complete model response with no fixed structure, field-by-field masking cannot cover every risk. Yet an investigation may still need the original text. This example uses envelope encryption to show how data transformation can first derive the analysis field msg_len from the response length, then encrypt the sensitive body and write it to a protected LogStore:

* | extend msg_len = length("gen_ai.output.messages")
  | extend result = envelope_encrypt(
      cast("gen_ai.output.messages" as binary),
      '-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0Vh1QIpNurP0bB9uHRK3
ZL29SjY74OtYa5nmf9xUMKAuLrio5BN6LRZyG680uCkCThCMOK/T9ka5ZYtEIu8b
U7acNR1dL3LWsbZG3ScbqyISJ3pEBli2rCgOqBQK3EE+6HGavh6wo8FAhkSBiryU
eVKxSJSnC928RvlBWtyfdubVYQHPONhx0TbCj38Njjfi8kbKGgE92q9gTlL+spge
qvgDec9GV2VzxhLoTgxd6jdHYyfFzuQucb0hZtmWqus4E4MTnAXnUJOGizmI02AP
5d49P44kZXElBPRw4A0WISKBmwOzYCZqPSEWyHnj/qIaXAbz90B8TCUiPZlNouCG
TwIDAQAB
-----END PUBLIC KEY-----',
      '{"mode":"RSA"}',
      '{"mode":"CTR"}')
  | extend encrypted_dek = result['encrypted_dek'],
           enc_output_messages = result['encrypted_data']
  | project-away "gen_ai.output.messages"

In the record written to the protected LogStore, gen_ai.output.messages is removed. It is replaced by enc_output_messages, which contains the ciphertext, and encrypted_dek, which contains the public-key-encrypted DEK. The remaining metadata stays in plaintext:

The protected LogStore retains fields such as event type, session ID, model, user ID, and response length, so teams can still use it for alerts, trend analysis, and risk investigations. The model response itself is encrypted. For deeper investigations, authorized security personnel can decrypt selected records with the private key.

This strikes a balance between encryption and analyzability: rather than encrypt every field, extract the metadata needed for analysis first and encrypt only the highly sensitive body. Routine analysis uses the metadata, while the original sensitive content can be accessed only through authorized decryption.

How to Choose: Plaintext Boundary, Analysis Needs, and Decryption Permissions

Ask three questions when choosing an approach.

First, can plaintext leave the collection source? If not, process it at the source.

Second, must the logs remain readable for routine analysis? If so, preserve readability. Encrypt only when the content is highly sensitive as a whole and cannot be reliably protected field by field.

Third, who is allowed to decrypt the data? One team within a single security domain can manage keys centrally. Across teams, or when duties must be separated, decryption authority should remain under the independent control of the security or audit team.

The following table summarizes the options:

Scenario	Recommended processing stage	Recommended method
AI gateway logs on hosts or in containers; plaintext must not leave the host	LoongCollector on the client side	Data masking with mask; encrypt selected fields as needed
SDKs or third-party systems writing directly to SLS	Ingest processor	Masking or encryption before ingestion
Historical logs that require protection	Data transformation	Transform data from the source LogStore, then write masked or encrypted output to a new LogStore
Internal RAG Q&A content that requires controlled recovery	Client-side processing or ingest processor	Symmetric encryption with aes_encrypt
Entire blocks of highly sensitive content, such as tool output or configuration files	Client-side processing or data transformation	Envelope encryption with envelope_encrypt

Keep Logs Secure Without Losing Their Value

Logs make systems observable and support analysis and traceability. Sensitive data protection should preserve that value within a clear security boundary. SLS applies masking and encryption during collection, ingestion, and transformation, helping organizations reduce plaintext exposure while retaining the analysis they need.

Organizations can choose masking or encryption based on the data type. They can apply protection during collection, before ingestion, or after ingestion through data transformation, then choose symmetric or envelope encryption based on who controls the keys. Together, these choices define a clear, enforceable boundary that keeps data both useful and secure.

Alibaba Cloud Partners with Datadog Close Go's Last Observability Gap

ObservabilityGuy — Wed, 09 Sep 2026 02:13:59 +0000

This article introduces OpenTelemetry Go Compile-Time Instrumentation v1, enabling zero-code, build-time observability for Go applications.

Go was the last mainstream language without zero-code observability. In July 2026, the OpenTelemetry Go Compile-Time Instrumentation project shipped its stable v1 release. Launched jointly by Alibaba and Datadog and developed through a year and a half of community collaboration, it lets Go developers add distributed tracing and metrics collection to an application with a single command, without touching any business code.

This article walks through how this milestone works, how to use it, and when to choose it.

1. Why Go Has Long Lacked Zero-Code Observability

Java has -javaagent, Python has sitecustomize, Node.js has --require.NET has CLR Profiler. AI Agents in these languages inject agents dynamically at runtime, so developers get traces and metrics without changing a line of code.

Go can't do that, and the reason is fundamental: Go compiles to a static binary — no VM, no bytecode, no class-loading hooks. Once go build finishes, you have a self-contained machine-code file with nowhere to "attach" an agent at runtime.

That has long left Go developers with two options:

Manual instrumentation: call span.Start() / span.End() explicitly in every HTTP handler and around every DB call — intrusive, and easy to miss spots
eBPF observation from the outside: capture network calls in the kernel — zero-touch, but you see only L4/L7 protocol information, with no business semantics

A wide gap separates the two. For a platform engineering team running hundreds of Go microservices, "add tracing by hand to every service" isn't realistic. eBPF covers a lot of ground but can't reach inside the code — if you want to know which SQL query was slowest in a request, eBPF can only tell you "this TCP connection took 200 ms"; it gives you nothing at the database/SQL level.

Compile-time instrumentation fills exactly that gap: probes are injected during go build, and the resulting binary carries observability with it. No code changes, no external AI Agent.

2. Compile-Time Instrumentation: Injecting Agents at Build Time with -toolexec

The Go toolchain has a little-known but very powerful extension point: -toolexec.

When you run go build, the go command is really orchestrating lower-level tools such as compile and link. -toolexec lets you name a wrapper program, and the Go toolchain routes every invocation of compile/link through that wrapper first — much the way Unix strace or time wraps a command.

OpenTelemetry Go Compile-Time Instrumentation builds on this mechanism. Its core tool, otelc, acts as a wrapper around -toolexec. As the compiler processes each package's source, otelc does the following:

Parse the AST: read the current package's abstract syntax tree
Match rules: check whether a registered instrumentation rule applies (for example, "this is ListenAndServe in net/http")
Inject code: weave tracing/metrics code into the target function before compilation
Pass through to the compiler: hand the rewritten source to the original compile for normal compilation

The binary you end up with has the agent code built in. At runtime there is no extra agent process, no sidecar, and no eBPF program to load.

How this fundamentally differs from a Java agent: Java injects logic at runtime by rewriting bytecode, which incurs runtime overhead (every class load passes through a transformer). Go compile-time instrumentation does all the rewriting during the build, so there is no additional runtime overhead — what you get is an ordinary Go binary that happens to contain tracing code.

This is also what makes the approach so CI/CD-friendly: in your build pipeline you replace go build with otelc go build. No changes to your deployment architecture, no Pod spec edits, no DaemonSet.

3. What v1 Supports: Coverage from net/http to gRPC

As the first stable release, v1 focuses on the most heavily used library categories in the Go ecosystem:

Library	Type	Signals produced
net/http	HTTP client/server	Trace spans + HTTP semantic attributes (method, status_code, route)
Database/SQL	Database access	DB spans + statement summary + connection information
google.golang.org/grpc	RPC framework	Client/Server spans + gRPC semantic attributes
github.com/redis/go-redis	Redis client	Redis command spans
Go runtime	Runtime metrics	GC, goroutine, memory metrics

A few key design decisions:

Rule-based architecture. The instrumentation logic for each library is defined as a rule, and a rule describes the package paths and function signatures to match plus the code template to inject. This means the community can contribute new rules independently, without touching the core framework. After v1, supporting a new library essentially means submitting a new rule.

Semantic Conventions compliance. Every span and metric produced follows the OpenTelemetry Semantic Conventions — attribute names, span names, and metric units are fully standardized. Whatever backend you use (Jaeger, Tempo, Simple Log Service, ARMS), the data means the same thing.

Automatic discovery. By default, otelc scans your go.mod dependency tree and automatically finds and enables every library covered by a registered rule. You don't have to declare "instrument net/http" — if you use it, it gets instrumented.

v1 deliberately prioritized a focused, high-quality core over breadth. Every supported library has been through full correctness tests and performance benchmarks. Later releases will keep expanding coverage.

4. Get Started in 3 Minutes: Add Tracing with One Command

Install otelc:

go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest

Option 1: replace the build command directly

otelc go build -o myapp .

That's it. The resulting myapp already has tracing code built in. Set OTEL_EXPORTER_OTLP_ENDPOINT at startup and trace data will flow to your Collector automatically.

Option 2: leave the build command unchanged (CI/CD friendly)

otelc setup 
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'" 
go build -o myapp .

This fits better when you already have a complex Makefile or CI pipeline — you add two lines of setup to the build environment and leave the existing go build command alone.

Dockerfile integration example:

FROM golang:1.23 AS builder
RUN go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest
WORKDIR /app
COPY . .
RUN otelc go build -o /myapp .
FROM gcr.io/distroless/base
COPY --from=builder /myapp /myapp
ENTRYPOINT ["/myapp"]

Image size is unaffected: otelc is only used during the build, and the final image contains only the compiled output.

5. Choosing Among Three Paths: Compile-Time vs. eBPF vs. Manual

Go observability now has three complementary paths, not competing ones:

Dimension	Compile-time instrumentation	OpenTelemetry eBPF Instrumentation (OBI)	Manual instrumentation (Go API)
Prerequisites	Able to recompile the source	Linux kernel ≥ 4.x, privileges	Able to modify the source
Code intrusion	None	None	High
Runtime overhead	Very low (code is inlined)	Low (collected in kernel space)	Depends on the implementation
Depth of coverage	Function level (including third-party dependencies)	Protocol level (HTTP/gRPC/SQL)	Any granularity
Multi-language support	Go only	Go/Java/Python/Node.js, etc.	Go only
Deployment changes	Change the build command	Deploy a DaemonSet	Change code + redeploy
Best fit	Platform engineering teams rolling out observability uniformly	Existing services, multi-language clusters	Business logic that needs custom spans

How to decide:

If you can rebuild your services and want zero-code coverage with low overhead → compile-time instrumentation
If you have many existing services, don't want to recompile them one by one, or run a mixed-language cluster →OBI
If you need custom spans inside specific business logic (business semantics such as "user places order") → manual instrumentation

You can combine all three. Compile-time instrumentation covers the generic spans from the standard library and third-party dependencies, manual instrumentation adds business-semantic spans, and the two sets of spans stitch into one trace automatically. eBPF then serves as the fallback for older services you can't recompile yet.

In practice, for a typical Go microservice cluster the most pragmatic strategy is this: use compile-time instrumentation plus a little manual instrumentation for new services; cover existing services with eBPF first, then move them to compile-time instrumentation in CI over time.

6. Community Collaboration and Next Steps

How this project came about is itself an interesting case of open-source collaboration.

In early 2025, Alibaba and Datadog were each exploring Go compile-time instrumentation internally and discovered each other's work. Rather than build separately and then fight over the standard in the community, the two companies merged their efforts into the OpenTelemetry community and created a dedicated Special Interest Group (SIG) under CNCF to drive the work in a vendor-neutral way.

Over a year and a half, the project went all the way from PoC to stable. Key milestones:

SIG formed (2025 Q1): technical direction and governance structure settled

Core framework landed (2025 H1): rule engine, AST rewriting, test infra

Community growth (2025 H2): new contributors joined through CNCF LFX Mentorship — among them Azhar Momin, who went from mentee to approver

v1 released (2026 Q3): first stable version, covering 5 core library categories

What's on the roadmap next:

More instrumentation rules: Kafka, MongoDB, AWS SDK, and others
Registry integration: discover and install community-contributed rules through the OpenTelemetry Registry
Build performance optimization: reduce otelc's impact on compile time
Validation in more scenarios: production case studies and performance benchmarks

7.Final Thoughts

Go's observability gap is finally closed.

If you're a platform engineer running observability infrastructure for dozens or hundreds of Go services, otelc go build may be the highest-ROI change you can make: one command, full coverage, nothing intrusive.

If you maintain a library or are simply interested in OpenTelemetry, we'd welcome your rule contributions. Writing an instrumentation rule for a library is far easier than implementing an SDK wrapper from scratch — a rule is essentially a declarative description of what code to inject, in which function, at which position.

Resources:

Project repository: github.com/open-telemetry/opentelemetry-go-compile-instrumentation
Quick-start docs: Getting Started in the project README
CNCF Slack channel: #otel-go-compile-instrumentation
OBI: github.com/open-telemetry/opentelemetry-go-instrumentation

STAROps Synthetic Monitoring Intelligent Analysis in Practice: From Task Analysis to Cross-Domain APM Fault Demarcation

ObservabilityGuy — Fri, 04 Sep 2026 05:46:11 +0000

This article explains how STAROps integrates Synthetic Monitoring, SLS, UModel, and APM to automatically analyze external access anomalies and accurately pinpoint cross-domain faults.

When a user reports that "the site won't load" or "the API suddenly got slow", network monitoring usually flags the anomaly quickly. The hard part comes next: did the fault occur on the network path between the user and the server, or inside the server-side application? Does it hit one region or ISP only, or every user? Should you call the network team, the application team, or the owner of a downstream dependency?

Synthetic Monitoring is the tool that watches this external access experience. Probe nodes spread across regions and ISPs follow fixed rules to simulate real users visiting a website, an API, or a domain name, and they record data for each stage: DNS resolution, TCP connection, TLS handshake, and server response. Traditional synthetic monitoring reports, however, are better at showing what happened than at explaining why it happened.

STAROps brings intelligent O&M analysis to observability data. Its Agent automatically queries task configuration, synthetic monitoring logs, and application call chains, then links evidence that used to sit in separate systems. In the end it answers four questions: where the anomaly occurred, which user paths it affected, which side owns the problem, and what data backs the conclusion.

This article is for product managers, developers, and O&M engineers who are new to Synthetic Monitoring intelligent analysis. It explains why STAROps connects Synthetic Monitoring, SLS, UModel, and APM, and how that chain moves you from spotting an anomaly to explaining it and pointing at a fix. By the end, you will understand where traditional synthetic monitoring analysis breaks down, how STAROps assembles evidence across systems, and how a network problem is told apart from a backend one.

Getting Started: Key Concepts in the Analysis Chain

Synthetic Monitoring: Probe nodes actively visit a target website, API, or domain name and continuously check availability and performance from the user's point of view.
STAROps Agent: The intelligent analysis unit that queries data, aggregates evidence, judges faults, and generates reports. Referred to as the Agent below.
SLS: Alibaba Cloud Simple Log Service, which stores the raw execution record of every synthetic check. Think of it as the log warehouse you search for factual evidence during analysis.
UModel: A unified model that describes observable objects such as tasks and applications, plus the relationships between them. Think of it as a map that tells the Agent who each object is, where its data lives, and how the objects connect.
APM: Application Performance Monitoring, used to watch how a request executes after it reaches the backend. One complete request is a Trace, and each processing step within it is a Span.
Workspace: An isolated scope of observability data and resources. The Agent must pin down the Workspace before it can locate the right task, logs, and APM data.

1. Background: Gaps in Traditional Synthetic Monitoring Analysis

Figure 1: Main gaps in traditional synthetic monitoring analysis

Traditional analysis usually starts with a synthetic monitoring report: check availability rate, response time, and error codes, then switch by hand between logs, task configuration, and APM pages to look for a cause. These metrics work as a health overview, but they rarely add up to verifiable root cause evidence, so four gaps remain in troubleshooting.

First, task context is missing. The same error code can mean completely different things under different task types, target addresses, DNS configurations, assertion rules, and monitoring point coverage. Repeating the error code alone tells you nothing about whether the cause is a misconfiguration, a broken target service, or a failure at one protocol stage.

Second, raw evidence is missing. OpenAPI is the standard query interface the system exposes, and central-side metrics are already aggregated statistics; both suit stable queries and health overviews, but neither carries all the detail an ad-hoc drill-down needs. Stage durations, city and ISP, target IP, CNAME, raw DNS answers, response headers, assertion results, and Trace context usually live only in the raw synthetic monitoring records in the user's SLS.

Third, protocol stages are not opened up. A single HTTP check may run through DNS, TCP, TLS, request send, first-packet wait, HTTP response, response download, and content assertion. Look only at total duration and the final error code, and problems from different layers blur together.

Fourth, synthetic monitoring and the backend application are not linked. Synthetic Monitoring can tell you that external access failed or slowed down, but once the request enters the backend, the problem may sit in business logic, the database, the cache, or a downstream dependency. In APM, the server span marks where the backend receives and handles the request, and child Spans record internal steps such as database, cache, and downstream calls. Without that evidence, the Agent cannot reliably pinpoint a problem inside the backend.

Synthetic Monitoring intelligent analysis therefore needs a complete chain: task identification → data location → raw sample query → field filtering and combined analysis → protocol-stage drill-down → link diagram rendering → cross-domain APM alignment → final fault demarcation.

2. Core Capabilities

Figure 2: The five core capabilities of STAROps Synthetic Monitoring intelligent analysis

1. Synthetic Monitoring Task Analysis

Every synthetic monitoring task carries a unique taskId. Once you supply a taskId, a task name, or a target address, the Agent locates the matching synthetics.task entity — the structured description UModel keeps for that task. It then reads the task type, target, probe frequency, monitoring points, timeout, assertions, and Trace configuration, and queries raw synthetic monitoring samples within the specified time range.

The analysis does more than compute sample count, success rate, availability rate, response time, and error distribution. It also filters, aggregates, and compares by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Even when a task looks stable overall, single-task analysis still checks protocol stages, localized slow points, and configuration risks instead of simply reporting "running normally".

Coverage today includes HTTP/HTTPS, Ping, TCP, UDP, DNS, DNSTRACE, SMTP, POP3, FTP, Traceroute, MTR, API, Multi, Browser, WebSocket, and SSL. Each task type follows its own analysis path: Ping focuses on packet loss and RTT, API and Multi on failed requests and steps, Browser on the main document, page timing, and the resource waterfall.

RTT (round-trip time) is the time data takes to travel from the probe to the target and back. API, Multi, and Browser correspond to API requests, multi-step tasks, and browser page loads respectively.

2. Intelligent Workspace Inspection

Single-task analysis answers "why is this task failing"; Workspace inspection answers "which tasks in this group deserve attention first". A Workspace is an isolated observability space that holds the tasks, logs, and related resources belonging to one user or business scope.

Workspace inspection first rolls up all synthetic monitoring tasks into an overall health view, then drills down into the tasks that matter. Task selection weighs status severity, failure count, availability rate, latency, error concentration, failing-stage concentration, and how long the anomaly has lasted, so the ranking never rests on a single metric.

Inspection supports three cycle scenarios:

High-frequency monitoring: looks at the last 30 minutes for wide-scale unavailability, clustered timeouts, and high-priority risks, and analyzes the Top 1–3 tasks in depth.
Daily diagnosis: analyzes the last 24 hours against the same window the previous day, and reports trends, lingering risks, and priority tasks.
Weekly review: analyzes the last 7 days against the previous week, and identifies persistent anomalies, repeated fluctuations, recovered risks, and administration items.

3. Fine-Grained Protocol Diagnosis

The Agent picks protocol evidence to match the task type rather than explaining every task with one set of metrics:

HTTP/HTTPS: analyzes stage by stage along DNS → TCP → TLS → request send → first-packet wait → HTTP response → response download → assertion.
DNS: checks the query object, DNS Server, RCODE, CNAME, resolved IP, SOA, expected match, and whether the domain name exists.
WebSocket: additionally checks whether the HTTP Upgrade completed.
API and Multi: pinpoint the failing request or step.
Browser: separates main-document, resource-loading, page-performance, and operation-step anomalies.

If network protocols are new to you: DNS translates a domain name into an IP address, TCP establishes a reliable connection, and TLS encrypts the traffic and validates the certificate. RCODE is the result status returned by the DNS server, CNAME marks a domain alias, and SOA records the basic information of the domain's authoritative zone.

Diagnosis covers configuration problems as well as data problems. A DNS query object mistakenly set to a full URL, an HTTP 200 whose content assertion fails, a disabled task, a query window shorter than the probe interval, or no matching samples in SLS — each is flagged on its own and never lumped in as a target service anomaly.

4. Request Link Visualization

The synthetic monitoring access link diagram is generated dynamically from real samples. The first node is always the probe, and only the protocol stages the request actually reached appear after it. If the request fails at DNS, the diagram stops at DNS and does not draw TCP, TLS, or the backend service.

The diagram follows a "stage node → anomaly node" structure. A DNS anomaly, for example, renders as "probe → DNS resolution → resolution anomaly (NXDOMAIN)" rather than jumping straight from the probe to the anomaly. Normal nodes, affected paths, and the final root cause use different styles, so an anomaly propagation path is not mistaken for the root cause.

Figure 3: Synthetic monitoring access link diagram generated from the actual execution stages in STAROps task analysis

The diagram does not depend on APM. With no Trace available, it still shows the synthetic monitoring access path; when a backend Trace exists, an APM call chain diagram is added to show the backend entry point, API, business processing, downstream dependencies, and the specific point of failure. The trace_id ties the two diagrams to the same request.

5. Cross-Domain APM Fault Demarcation

When an HTTP, API, Multi, or Browser task carries Trace evidence such as trace_id, traceparent, or traceInfo, the Agent picks a failed or high-latency sample and queries the APM Trace itself, instead of merely telling you to look it up.

A Trace is the full path of a request from the probe into the backend and on to other components; a Span is one step along that path. Linked analysis aligns four kinds of Span:

Synthetic monitoring client span: the external view, from the probe sending the request to receiving the response.
Backend server span: the entry view, from the request reaching the application to the application returning a response.
Local business Span: the execution of a Handler, business method, or internal computation.
Downstream dependency Span: a call to a database, cache, RPC, or external HTTP service.

By comparing synthetic monitoring stage durations, the server span, child Spans, and the first concrete error, the Agent can place the fault on the network path, the path before the application entry, the backend service, or a downstream dependency — or state that evidence is insufficient. A 502 from a backend API may be only the surface symptom; if the first concrete error appears in a downstream DNS, database, or external service Span, the conclusion moves to the downstream dependency.

3. Synthetic Monitoring Architecture and Technical Approach

Figure 4: Analysis architecture combining UModel, SLS, and APM

Data Organization: UModel and SLS Working in Two Layers

Think of UModel as the map and SLS as the field notes. UModel holds relatively stable object information: which task this is, what it targets, which Workspace it belongs to, where its data lives, and how it relates to other objects. SLS holds the execution facts that keep arriving: which probe hit which target at what time, which protocol stages it went through, and whether it succeeded.

Creating a synthetic monitoring task produces a synthetics.task entity in UModel that holds metadata such as taskId, task type, target address, probe configuration, Workspace, SLS data coordinates, and Trace configuration. Every real probe run then writes one or more execution records into SLS. One task entity therefore maps to many synthetic monitoring samples; each probe run does not create a new entity.

When a request arrives, the Agent first queries this map through UModelSearch to locate the task entity and its log location, then enters the matching SLS Project and Logstore to query samples by taskId and time range. A Project is the isolated resource space in SLS, and a Logstore is the data container inside it that holds one class of logs.

Analysis Strategy: Overall Aggregation, Risk Screening, and Protocol Drill-Down

With the samples in hand, the Agent first sizes up the scope and distribution of the anomaly, then drills down step by step into specific protocol stages. Single-task analysis aggregates by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Workspace inspection instead rolls up all tasks first, then screens out the ones with low availability rate, high latency, concentrated errors, or persistent anomalies.

After screening, the Agent reads representative failed or high-latency samples and takes a different analysis path per task type. An HTTP task, for instance, is checked in order for DNS, TCP, TLS, request send, first-packet wait, response download, HTTP status code, and content assertion; a DNS task goes on to read RCODE, the number of answer records, CNAME, resolved IP, SOA, DNS Server, and the match result.

This "overall aggregation → risk screening → protocol drill-down → representative sample" approach separates resolution failures, connection problems, certificate issues, slow server-side processing, and content assertion failures, instead of explaining only the final error code.

Cross-Domain Linkage: From the Synthetic Monitoring Domain into the APM Domain

The analysis chain can continue into APM only when the synthetic monitoring sample carries Trace information. For any task with Trace enabled, the system generates a synthetic APM service and uses the same_as relationship in UModel to map synthetics.task to that service permanently. For the Agent, this relationship is the signpost from the synthetic monitoring domain into the APM domain.

same_as answers "which synthetic APM object corresponds to this synthetic monitoring task"; trace_id answers "which applications and call nodes did this particular request pass through inside the backend". The first is a stable task-level relationship, the second a precise per-request association. Note one thing in particular: same_as does not equate the synthetic monitoring task with the real backend application.

Inside APM, the Agent looks at the synthetic monitoring client span, the backend server span, local business Spans, and downstream dependency Spans together:

If synthetic monitoring duration rises and the server span rises with it, the problem leans toward the backend service.
If most of the time sits in a database or downstream call, you can narrow it to that dependency.
If the client span is slow while the server span is short, the problem more likely lies in the network, CDN, gateway, queuing, or another uninstrumented segment.
If DNS, TCP, or TLS already failed and no server span exists, the request never reached the application.

If a sample has a trace_id but complete backend Spans are missing, the Agent explicitly marks "insufficient APM evidence" rather than reading "no data found" as "backend is normal".

Design Principle

The STAROps Agent does not hand a fixed report to a large model for summarizing. It builds a query plan first, then constrains its conclusions with real data. UModel serves as the semantic control plane that identifies objects and relationships, and the user-side Logstore serves as the data plane that holds observed facts: indexed fields screen samples fast, and SPL queries then fetch and process finer protocol details. Here, an index is a pre-built field for fast retrieval, and SPL is the query language used to filter, aggregate, and transform log data.

Different tasks load different evidence models, normalizing heterogeneous logs into five layers: task, single execution, protocol stage, blast radius, and call chain. When a sample carries Trace information, the Agent follows trace_id into APM and aligns the external client span with the backend server span, business Spans, and downstream dependency Spans. The output is not an empirical guess a large model made from field names; it is a traceable fault demarcation conclusion constrained by raw samples, aggregated results, protocol semantics, and cross-domain Traces.

End-to-End Analysis Chain

Parse the user input and decide whether the scope is a single task or a Workspace.
Query synthetics.task to obtain the task configuration and SLS data coordinates.
Fix the time window and calibrate time units.
Aggregate overall results and screen for risk using indexed fields.
For abnormal tasks, read representative samples and unindexed protocol fields.
Enter the dedicated analysis for DNS, HTTP, Browser, API, Multi, or another type as appropriate.
If a Trace exists, pick a failed or high-latency sample and query the APM call chain.
Align task configuration, synthetic monitoring stages, blast radius, and Span evidence, then output the fault demarcation conclusion, link diagram, recommended actions, and data limitations.

The core strategy is "overall aggregation → risk screening → protocol drill-down → representative sample → cross-domain alignment". It keeps query volume under control across long time windows and large Workspaces while preserving the raw evidence needed to pinpoint the failing stage and the root cause.

4. How to Use

For task analysis, simply supply a taskId, a task name, or a Workspace, for example:

Analyze the synthetic monitoring tasks over the last 30 minutes.

You do not have to write log queries yourself. The Agent turns a natural-language question into concrete steps: task location, sample query, protocol analysis, and APM linkage.

The Agent then completes the following steps automatically:

Identify the scope and confirm whether this is single-task analysis or Workspace inspection.
Find synthetics.task through UModelSearch or the task query capability.
Locate the user-side SLS Project and Logstore.
Query raw synthetic monitoring samples within the fixed time window.
Run protocol-stage analysis according to the task type.
If Trace evidence exists, go on to query the APM Trace.
Output the complete Synthetic Monitoring intelligent analysis report.

For ongoing inspection, choose the high-frequency monitoring, daily diagnosis, or weekly review template. The inspection scans the synthetic monitoring tasks in the Workspace on a schedule, screens out risky tasks, and generates a report automatically.

A "Long-Running Task" in STAROps is an analysis task that repeats automatically on a schedule; a "Digital Employee" is an Agent pre-configured with skills, tools, and data permissions. Combine the two and synthetic monitoring inspection runs on schedule and pushes its results to the people you designate.

Synthetic Monitoring Task Analysis

Figure 5: Starting a synthetic monitoring task analysis in the STAROps console

In the STAROps console, you can ask about synthetic monitoring task details directly, for example "Analyze the synthetic monitoring tasks over the last half hour", "Analyze why a given synthetic monitoring task is failing", or "Show the synthetic monitoring tasks whose availability rate dropped sharply in the last day".

Synthetic Monitoring Inspection Tasks

Figure 6: Opening Long-Running Task in STAROps

Click "Long-Running Task" in the upper-left corner of the STAROps console.

Figure 7: Creating a synthetic monitoring inspection task and configuring the Workspace

Click "Create Task" in the upper-right corner of the page, select a Digital Employee and the Workspace the task belongs to, and enter a prompt such as "synthetic monitoring inspection".
Select one of the three synthetic monitoring inspection templates or combine them, and describe the inspection details in text; by default the inspection runs at Workspace level.

Figure 8: Configuring notification recipients and adding the job to Long-Running Task

4. Click "Configure Now" to add notification recipients, then click "Add to Long-Running Task" once you have made your selection.

5. Confirm the configuration is correct, then click "Confirm Execution".

Figure 9: Viewing the inspection report generated by the Long-Running Task

6. Click "Report" at the top of the page to view the detailed inspection results.

5. Fault Demarcation Logic

1. Locate the Layer Where the Request Stopped

An HTTP call chain runs DNS → TCP → TLS → request send → first-packet wait → HTTP response → download → assertion. The deepest stage the Agent can observe tells it whether the request reached the service entry point.

A DNS failure with no TCP evidence, for example, means the request never established a connection. An HTTP 502 shows that DNS resolution, the TCP handshake, and the response header return all completed, so the problem cannot be a first-packet wait timeout. An HTTP 200 with a failed assertion means the service is reachable and the failure sits in the response content or the business logic.

2. Determine the Scope of Impact and Temporal Pattern

Simultaneous failures across multiple cities and ISPs point to the target service, a public entry point, or a cross-network path. Concentrated failures in a single city or ISP suggest a regional network or ISP link issue. If only some monitoring points fail within the same city and ISP, the Agent drills down by client_id, DNS Server, local DNS, and egress path when these fields are available.

The Agent also separates continuous, sudden, intermittent, and recovered anomalies over time. It never generalizes a single spike into a continuous failure, and averages never stand in for P95, maximum values, or anomalous time buckets.

3. Prioritize DNS Evidence Over Outer Error Codes

The DNS error_code reflects the probe's local judgment, while RCODE is the protocol-layer response from the DNS server — the two are not interchangeable.

When raw_exception is present, the Agent reads rcode, rcode_name, and the answer record count first, to tell NXDOMAIN, SERVFAIL, REFUSED, FORMERR, NOTIMP, and NODATA apart (NODATA means RCODE is NOERROR but the target record is empty). If those fields are missing, it draws evidence from dns_raw, dns_raw_inner, message, SOA, EDE, and DNSSEC information, and states the confidence level of the conclusion explicitly.

DNS analysis also compares the DNS Server, city, ISP, monitoring point, CNAME, resolved IP, and target edge zone. Note that if error_code=0 but only an SOA record is returned without A, AAAA, or CNAME records, this only indicates that the current rule evaluation passed; it does not prove that the target record resolved normally.

4. APM Evidence Determines the Backend Boundary

If total synthetic monitoring duration rises while DNS, TCP, and TLS stay normal, and the server span rises with it, the problem lies in the backend service.
If a specific business sub-Span within the server span accounts for most of the duration, the application's internal processing is slow.
If the first concrete error, or most of the duration, appears in a downstream Dependency Span, the problem lies in that downstream dependency.
If the client span is markedly slow while the server span is very short, the problem more likely lies in the path before the application entry, the gateway, the CDN, or an uninstrumented component.
If DNS, TCP, or TLS fails and there is no server span, the request never entered the application.
If a trace_id exists but APM has no data or the Spans are incomplete, the evidence is insufficient, and you cannot conclude from it that the backend is healthy.

The gap between the client span and the server span is not pure network time; it may also include time spent in the CDN, gateway, ingress proxy, queueing, and uninstrumented components. The final conclusion therefore has to draw on synthetic monitoring stage fields, target access evidence, and Span parent-child relationships together.

6. How Cross-Domain APM Analysis Works

Figure 10: Analysis flow from a synthetic monitoring task into the APM call chain

A full cross-domain APM analysis takes four steps:

Starting from the synthetic monitoring task, the Agent uses UModelSearch to locate the synthetics.task entity, confirming the task type, target address, Workspace, and data coordinates.
The Agent queries the synthetic monitoring SLS samples to complete the analysis on the synthetic monitoring side; if the samples carry Trace evidence, it extracts a representative trace_id.
The Agent uses the trace_id to query the APM Trace, retrieving the server span, business processing Span, and downstream Dependency Span for the same request within the backend application.
The Agent aligns the evidence from the synthetic monitoring side and the APM side to form a cross-domain fault demarcation conclusion.

Evidence alignment follows these rules:

Synthetic monitoring is slow and the APM server span slows with it: the problem most likely lies in the backend service.
Synthetic monitoring is slow and the APM downstream Dependency Span slows with it: the problem most likely lies in a backend downstream dependency.
Synthetic monitoring is slow but the APM server span is very short: the problem leans toward the network link, the ingress layer, the gateway, or the path before the application entry.
The DNS, TCP, or TLS stage fails and there is no server span: the request never entered the application, so look first at the path before the application entry.
Trace evidence exists but no backend server span can be found: you cannot conclude that the backend is healthy. The Agent can only flag "insufficient APM evidence" and check Trace reporting, sampling, Region, Workspace, and APM integration.

The APM linkage presents evidence as two link diagrams: one shows the external synthetic monitoring access path, the other the backend Trace path, and trace_id aligns them as the same request. Together they show whether the request reached the service entry point and, once it was inside the application, exactly which Span failed.

Figure 11: Aligning the synthetic monitoring access path with the backend APM Trace via trace_id

7. From "Seeing the Anomaly" to "Knowing What to Do Next"

The value of STAROps is not longer reports; it is a shorter distance from alert to action. It organizes task configuration, raw SLS samples, monitoring point distribution, protocol stages, and APM Traces into one continuous chain of evidence. Users can start from a taskId, task name, or Workspace and work out step by step whether an anomaly comes from a global outage, a regional network, a single ISP, a specific monitoring point, the task configuration, or the backend application and its downstream dependencies.

For experienced engineers, these diagnostic capabilities cut down the constant switching between the Synthetic Monitoring console, SLS, and APM to piece evidence together by hand. For anyone handling a synthetic monitoring fault for the first time, they turn an expert's troubleshooting sequence into a repeatable analysis flow. At Workspace level, the Agent can also pick high-risk targets out of a large pool of tasks and keep running high-frequency monitoring, daily diagnosis, and weekly reviews.

Appendix: Glossary of Terms

Term	Definition
Synthetic Monitoring	Simulates user access from distributed probe nodes to proactively check the availability and performance of target services.
STAROps Agent	An intelligent analysis unit responsible for invoking tools, querying data, aggregating evidence, and generating diagnostic conclusions.
Workspace	A mutually isolated set of observable resources and data scopes.
taskId	The ID that uniquely identifies a synthetic monitoring task.
UModel	A unified model describing observable entities and their relationships, used to locate analysis targets and navigate across domains.
SLS	Alibaba Cloud Simple Log Service (SLS), used to store and query synthetic monitoring execution samples and related logs.
Project / Logstore	A Project is a resource isolation space in SLS, and a Logstore is a data container within it that stores a specific category of logs.
SPL	A query language used to filter, aggregate, and transform log data.
synthetics.task	The entity type in UModel that represents a synthetic monitoring task.
APM	Application Performance Monitoring, used to observe the execution of requests after they enter the backend application.
Trace / Span	A Trace is the complete call chain of a single request, and a Span is an individual processing step within that chain.
same_as	The UModel relationship that associates a synthetic monitoring task entity with its corresponding synthetic monitoring APM service.

Conclusion

Synthetic Monitoring captures the external user experience before a request reaches the service, APM observes how the request executes once inside the application, SLS preserves the factual evidence, and UModel locates the objects and connects the different data domains. STAROps organizes these four into a continuous chain that runs from task discovery through sample aggregation and protocol diagnosis to backend fault demarcation.

This practice turns Synthetic Monitoring from a system that only displays abnormal metrics into an analysis entry point that explains anomalies and supports decisions. Instead of a raw list of fields, users get verifiable answers: where the request stopped, who the anomaly affected, which side owns the fault, what the evidence is, and what to tackle first.

From Reinventing the Wheel to Embedding STAROps via OpenAPI: Why NIIMBOT Chose a Cloud AIOps Foundation

ObservabilityGuy — Fri, 04 Sep 2026 02:00:02 +0000

NIIMBOT integrated Alibaba Cloud's STAROps and UModel via OpenAPI into its SRE platform for automated, cross-domain AIOps diagnostics.

Background

When a technically mature team has moved its entire stack to the cloud and already built a complete observability stack, the key question is no longer "can we build it?" but "what should we build ourselves, and what should we hand to the cloud?" That was the decision facing NIIMBOT. With an established in-house SRE platform, should the company keep building its own observability data foundation and operations digital twin, or adopt a more complete set of ready-made capabilities?

NIIMBOT is dedicated to making things easier to manage through innovation. Since its founding in 2012, the company has continued to advance smart label printing hardware, cloud printing service platforms, and enterprise performance management systems, making label printing smarter and easier. NIIMBOT products are now used across retail, telecommunications, office settings, industry, healthcare and laboratories, and homes, reaching more than 22 million users worldwide.

Business Challenges

As the business expanded rapidly around the world, its systems grew exponentially in scale and complexity, pushing operational complexity to a new level. NIIMBOT runs its business systems on Alibaba Cloud and had already assembled a comprehensive observability stack: Real User Monitoring (RUM) for frontend experience data, Managed Service for Prometheus for container and cloud resource metrics, Application Real-Time Monitoring Service (ARMS) for application performance tracing, and Simple Log Service (SLS) for logs. A self-managed Grafana instance connected to these cloud data sources to provide unified dashboards. Even with this extensive toolset, three structural problems remained.

(1) No Clear Global Topology: Internal Calls and External Dependencies Sat in Separate Graphs

The business systems ran in the cloud, but one question remained hard to answer: which APIs does an application call, and which cloud resources does it depend on? ARMS showed some internal API call relationships, while RDS instances, Redis instances, message queues, and other external dependencies remained scattered across their own monitoring views. The two could not be combined into a single dynamic graph. As the business grew, mapping topology by hand became increasingly inefficient — dependencies documented today could be out of date after next week's release.

(2) Multi-Dimensional Observability Data Was Complete, but Disconnected

Metrics, logs, traces, and events were all being collected. The problem was that collecting everything is not the same as connecting everything. Troubleshooting an incident meant context-switching repeatedly among metric charts, traces, logs, and change events, matching timestamps by hand, with no single thread automatically linking the different data dimensions. The data was there; people still had to piece the clues together.

(3) Alert Noise Made Cross-Domain Root Cause Analysis Take Tens of Minutes

When an incident occurred, every component across the stack — frontend, backend, middleware, database, and containers — raised alerts at once, burying the true source of the problem. Root cause analysis depended heavily on experience: engineers had to decide by instinct where to begin, and locating and analyzing a cross-domain fault routinely took tens of minutes.

These challenges raised a question specific to NIIMBOT. The team was capable of building its own SRE platform, but it also understood the cost of automatically correlating multi-dimensional observability data and maintaining a dynamic, real-time view of topology. Both required substantial investment and continuous upkeep. Was this particular wheel really worth building in-house?

Solution: A Multi-Dimensional Data Foundation, a UModel Digital Twin, and OpenAPI Integration with the In-House SRE Platform

After technical discussions with Alibaba Cloud, NIIMBOT found that the Alibaba Cloud observability system already provided the two capabilities it needed most: automatic correlation across observability data dimensions and dynamic awareness of end-to-end topology. The correlation coverage was broad, and the topology updated automatically — exactly what the team had planned to build. NIIMBOT therefore decided to stop reinventing this wheel. Alibaba Cloud would provide the observability data foundation and operations digital twin; the in-house SRE platform would call STAROps diagnostics through OpenAPI and focus on orchestration and closed-loop workflows tailored to NIIMBOT's business. The solution was implemented in four layers.

(1) A Unified Multi-Dimensional Observability Data Foundation: Bringing Four Existing Collection Capabilities onto One Data Plane

Rather than replace the RUM, Prometheus, ARMS, and SLS deployments already in use, NIIMBOT used Cloud Monitor 2.0 (CMS 2.0) to collect, store, view, and analyze metrics, logs, traces, events, and changes in one place. Frontend experience data from RUM, container and cloud resource metrics from Prometheus, application performance and traces from ARMS, and business logs from SLS now converge on the same data plane. This provides a consistent foundation for topology modeling and intelligent diagnostics. In short, the data no longer sits in separate silos; it enters a shared correlation layer with consistent definitions.

(2) The UModel Operations Digital Twin: Automatic End-to-End Topology Modeling with Real-Time Updates

This capability is the main reason NIIMBOT shifted from building to adopting. UModel models business systems automatically and constructs an end-to-end topology spanning internal API calls and external cloud resource dependencies — frontend → backend → middleware → database → containers — in a single graph. The graph updates in real time as applications are deployed, dependencies change, or workloads scale, with no manual upkeep. Clicking any node brings up its associated observability data. The topology is not a static architecture diagram but a living map backed by current data. UModel's mature correlation coverage and dynamic topology awareness spare the team the ongoing work of building and maintaining these models itself.

(3) STAROps Intelligent Diagnostics: Automatic Cross-Domain Root Cause Analysis on the UModel Topology

With a unified data foundation and the UModel topology in place, STAROps can perform root cause analysis (RCA) automatically when an incident occurs. It follows upstream and downstream relationships in UModel, finds relevant nodes, retrieves their observability data, and quickly returns investigation steps and analysis results. What once required engineers to guess where to begin and query multiple systems by hand becomes AI-driven, cross-domain correlation. In practice, STAROps performs reliably on single-domain analysis of infrastructure resources, such as resource utilization levels for a Pod or an anomalous cloud resource metric. In more complex cases — frequent garbage collection in a Pod, for example, which spans the container layer and application-level JVM metrics — STAROps can start from application monitoring, follow the topology to the JVM metrics, and pinpoint the root cause. This is the capability that turns "the data is there, but people assemble the clues" into "AI connects the data dimensions automatically."

(4) STAROps Integrates with the In-House SRE Platform: Embedding Diagnostics via OpenAPI

NIIMBOT engineers no longer need to leave their familiar SRE platform for another console when troubleshooting. Through OpenAPI, the in-house platform calls STAROps directly as a diagnostic engine and receives a live stream of its analysis and diagnostic results. With one click inside their own SRE platform, engineers can follow the STAROps reasoning process and see its conclusions in real time, completing full-domain diagnosis without leaving the platform.

This model embeds capabilities instead of replacing platforms. NIIMBOT gains the full diagnostic power of the Alibaba Cloud observability system while preserving the autonomy of an in-house SRE platform built around its own business logic. The cloud provides the foundation; the customer's platform remains the interface.

Business Value: Let the Cloud Handle the Heavy Lifting and Refocus on the Business

(1) One-Click, Closed-Loop Full-Domain Diagnosis Inside the In-House SRE Platform

Previously, troubleshooting a cross-domain fault meant switching repeatedly among ARMS, Prometheus, SLS, and Grafana. Engineers matched timestamps and assembled clues by hand, and root cause analysis routinely took tens of minutes. Now, a single click in the NIIMBOT in-house SRE platform triggers diagnosis of a business system issue. STAROps follows the UModel topology, correlates multi-dimensional observability data across domains, and returns the root cause analysis. The operations team no longer has to sift through each data dimension or hunt for clues layer by layer.

(2) No More Reinventing the Wheel: A Mature Team Puts Its Effort Back into the Business

NIIMBOT once had to commit its own engineering resources to building and maintaining general-purpose operations capabilities such as topology modeling and multi-dimensional data correlation. That costly, maintenance-intensive work is now handled by the Alibaba Cloud observability system and UModel. Freed from reinventing the wheel, the team can focus again on cloud resource management and system architecture optimization — work that contributes more directly to its business. For mature teams, deciding what to build and what to hand to the cloud is a real strategic choice; NIIMBOT has made its own.

(3) Operations Evolves from Reactive Firefighting to Proactive Management

The operations team once spent its days waiting for alerts, chasing faults, and conducting post-incident reviews. With UModel's dynamically updated end-to-end topology and AI-driven analysis of multi-dimensional observability data, the team has moved from reactive response to proactive inspection and earlier problem detection. While keeping business systems stable, its role has evolved from incident responder to steward of system health, completing its transition to an AIOps operating model.

Looking Ahead: From Diagnostics That Work to Diagnostics That Know NIIMBOT Better

As STAROps becomes embedded in the in-house SRE platform, more core business systems will gain end-to-end topology modeling and intelligent diagnostics, extending the living map and one-click cross-domain diagnosis across the entire business. Each new system onboarded strengthens the AIOps capability of the NIIMBOT in-house SRE platform: wherever the business expands, AIOps coverage follows.

(1) From Trace-Guided Correlation to Automatic Analysis from Any Entry Point

STAROps already pinpoints root causes by correlating across domains along the UModel topology. The next step is to make the result independent of where an investigation begins. Whether an engineer starts from application monitoring or from an infrastructure resource such as a Pod or container, the system should extend upstream and downstream automatically and connect every observability data dimension. The goal is more seamless, intelligent cross-domain correlation, with root cause analysis that no longer depends on choosing the right entry point.

(2) Making the Embedded OpenAPI Integration Smoother

Streaming the diagnostic process back to the in-house SRE platform through OpenAPI is central to this integration. Guided by NIIMBOT's feedback from day-to-day use, the two sides will continue improving the responsiveness and completeness of the stream, making the analysis shown inside the platform more detailed and coherent — and turning embedded capabilities into a genuinely seamless experience.

Flutter RUM in Practice: Reconstructing the Full Story Behind an AI App's Wait

ObservabilityGuy — Tue, 01 Sep 2026 07:31:52 +0000

In the AI Era, When an App Keeps Spinning, Don’t Blame the API Just Yet

When users report that "the page keeps spinning," "nothing happens after I tap the button," or "the content froze halfway through loading" in a Flutter app, most engineers immediately check API latency, error logs, or crash stacks. But production problems rarely have a single cause. One wait can span a user operation, page state, a network request, on-device rendering, error handling, and calls to native platform capabilities.

Looking at only one type of log can easily lead to an incomplete conclusion:

Check only API logs and you may miss on-device rendering and blocked state updates.
Check only Dart exceptions and you may not see the tap or the network request that preceded them.
Check only page instrumentation and you may not be able to tell which step the user is actually stuck on.
Check only crashes or errors and you may not be able to reconstruct the full path leading up to the problem.

The Alibaba Cloud Real User Monitoring (RUM) Flutter SDK addresses these problems for Flutter apps. Through alibabacloud_rum_flutter_plugin, it collects context in the Dart layer — pages, network, action, LongTask, exceptions, resource snapshots, and custom business fields — and passes that data to the native RUM SDK for reporting and correlation. This article uses these implementation details to show how to turn a single user wait into a traceable, verifiable account of what happened in production.

To Reconstruct an AI App's Wait, Follow the Path Through Flutter's Layers

A typical Flutter interaction path breaks down roughly into the following stages:

The user taps a button 
-> Flutter Action detection 
-> business state machine update 
-> network request or local task execution 
-> response returns 
-> page content refresh 
-> list, rich text, or image rendering 
-> the page reaches a stable state

All the user sees is "it took a long time." What engineers actually need to answer is where the time went: before the tap, before the request, during the request, after the response, or in the on-device rendering stage.

Traditional troubleshooting struggles to answer this question because it has several blind spots.

Visibility gap	Production symptom	Why traditional methods fall short
Behavior gap	No response after tapping a button, repeated taps, accidental triggers	Manual instrumentation can easily leave gaps, and definitions are inconsistent
Network gap	Slow APIs, failed requests, retries on weak networks	With only URL and duration, it is hard to get back to the specific page and user session
Rendering gap	Stuttering on list refresh, slow rich text rendering, brief page freezes	Server-side logs show nothing about main Isolate pressure on the client
Exception gap	State machine exceptions, parse failures, null object errors	The Dart stack lacks the preceding action and resource context
Session gap	Many logs exist, but they cannot be connected into a complete picture of a single user session	Engineers must manually correlate logs by timestamp, device, app version, and page path

The point of Flutter RUM, then, is not to collect a few more SDK event types. It is to build a connected timeline around a real user's experience:

Only when these events land in the same RUM Session can engineers turn "the page keeps spinning" into a set of testable hypotheses.

Connect the Clues Across Layers: Bridge Collection from Dart to Native

A Flutter app's observability path is inherently cross-layer. The Dart layer knows about Widget, Route, Zone, Dio, and business state; the Android, iOS, and HarmonyOS native SDKs are better suited to platform-side reporting, network tracing configuration, and data delivery.

Structurally, the Flutter RUM SDK splits into three layers:

The Dart collection layer preserves Flutter semantics — Dart Zone exceptions, Route lifecycles, Widget taps, Dio requests, and main Isolate blocking. The native SDK handles the standardized events and platform-side reporting.

This division of labor pays off most in complex Flutter scenarios: the Flutter side keeps the semantics of what the user did, what state the page was in, and how content refreshed, while the native and RUM sides place that data into a single session view.

The First Clue Starts with the Tap: Did the Action Reach the Current Session?

"Nothing happens after I tap" is a common report from production users. It can have several causes:

The button is disabled, or the business state machine never advances to the next step.
The tap triggers a request, but the request fails at the network layer or is retried.
The request goes out, then a synchronous task blocks the main Isolate, so the page does not refresh in time.
An automated flow or a system task triggers a duplicate operation, corrupting business state.

The first step here is not to open the API logs. Instead, confirm that the user behavior actually reached the same session.

User behavior in Flutter is not a native button tap, and it is not a Web DOM click. An operation starts as pointer events and coordinates, so the SDK has to go back into Flutter's HitTest, Widget, Element, and RenderObject system to recover the semantics.

Automatic action detection is not enabled by calling start() alone. The app must wrap the target widget tree in AlibabaCloudActionCapture:

AlibabaCloudActionCapture(
  child: MaterialApp(
    navigatorObservers: [
      AlibabaCloudRUMNavigationObserver(enablePagePerf: true),
    ],
    home: HomePage(),
  ),
);

For critical business operations, do not rely on default control detection alone. Add business semantics through ActionAnnotation:

ActionAnnotation(
  description: 'Submit Button',
  attributes: {
    'screen': 'order_detail',
    'action': 'submit_order',
    'actor_type': 'human',
  },
  child: ElevatedButton(
    onPressed: _submitOrder,
    child: Text('Submit'),
  ),
);

An operation driven by an automated business flow does not necessarily produce a standard pointer event that Flutter can observe. When the app calls a method directly, triggers a background task, or runs a rules engine, the SDK cannot recover the full semantics from a tap alone; if the tap comes from system accessibility or a simulated coordinate tap, it may still look like an ordinary tap. We therefore recommend that the app explicitly report an Action or a custom event at critical points in the flow, and add context such as actor_type.

actor_type	Meaning	Typical scenario
human	Action performed by a human user	Tapping submit, refresh, back, or retry
automation	Automated business flow	Scheduled retries, batch processing jobs, rule-triggered jobs
system	Triggered by an automated system process in the app	Automatic recovery, background refresh, timeout retry

With that in place, RUM has more than a single tap to work with when you investigate "nothing happens after I tap." By correlating the action with business events and subsequent resource, LongTask, and error events, engineers can reconstruct who or what initiated the operation and how it executed.

Where Did the Request Go After the Tap? Resource Reconnects Network Activity to the User Session

When a user says "the page keeps loading," the API is not necessarily slow. Break it into at least these stages:

From the tap to the request going out.
From the request going out to the response coming back.
From the response coming back to the page finishing its update.
Whether the page becomes interactive after it updates.

The Flutter RUM SDK offers two entry points at the network layer: when you use dart:io directly, wrap the global HttpClient through HttpOverrides; when you use Dio, integrate through AlibabaCloudRUMDioInterceptor.

final dio = Dio();
dio.interceptors.add(
  AlibabaCloudRUMDioInterceptor(
    onProvideSnapshots: (requestOptions, response, error) {
      return ResourceSnapshots(
        requestHeaders: {
          'content-type': requestOptions.headers['content-type'] ?? '',
        },
        responsePayload: response?.data is Map
            ? {
                'code': response?.data['code'],
                'requestId': response?.data['requestId'],
              }.toString()
            : null,
      );
    },
  ),
);

For a Flutter app, network monitoring is valuable not because it records one request's latency, but because it puts the API request, the user operation, and the page state back on the same experience path. When a user says "the page keeps spinning," engineers need to know whether the request was even sent, whether the API timed out, whether the server-side path failed, whether the page updated promptly once the response came back, and on which page, which version, and in which session all of this happened.

Where business policies permit, RUM can correlate client-side requests with server-side traces, turning troubleshooting from reading one isolated API log into reconstructing the full path of a user's wait. Requests involving third-party domains, sensitive APIs, or user input must still comply with business security policies and data governance requirements.

Resource latency alone is not enough. For critical business paths, add business fields that help locate the problem:

Field	Suggested source	Purpose
flow_id	Session-level custom property	Links the requests and page events within a single business process
request_id	Resource extension property or server-side trace	Aligns client-side requests with server-side logs
business_stage	Custom event or log	Identifies the business stage in which the wait occurred
result_status	Custom event	Distinguishes success, user cancellation, network failure, and server-side exception
error_stage	Custom event or log	Distinguishes failure stages such as parameter validation, networking, server-side processing, and rendering

These are not built-in fields the SDK promises to collect by default; they are context we recommend you add for business troubleshooting. The SDK provides a stable foundation for collection, reporting, and session correlation. Business semantics still have to be designed by engineers around their own paths and compliance requirements.

The Response Is Back. Why Is the Page Still Stuttering? LongTask Reveals Client-Side Pressure

When a Flutter page refreshes its content, it may trigger state updates, rich text rendering, image loading, long-list diffing, or JSON parsing. If that logic consumes too many on-device resources, the user sees a page that stutters, while the server-side logs look entirely normal.

The Flutter RUM SDK watches for periods when the Dart main Isolate remains unresponsive. When text rendering, list updates, or complex layouts consume too many client-side resources, the SDK can record a LongTask event and place the blocking duration, affected page, and user session on the same timeline.

Engineers do not need to focus on the underlying detection algorithm here. What matters is preserving evidence when the server has returned the content but client-side rendering cannot keep up. Then, when a user reports that "the page froze," the investigation can move beyond the server and network APIs to Flutter's rendering and state-update path.

These events fill in what happens on the client after the server responds. Within a single session, for example, you might see:

Action: Submit
-> Resource: /api/order/submit 200
-> Custom: business_stage = render_result
-> LongTask: 236ms
-> LongTask: 410ms
-> View: OrderResultPage

At that point engineers can form a working hypothesis: the API response is not slow, but the main Isolate is under significant pressure while the page refreshes. From there, keep verifying against data volume, list length, rich text node count, device model, and page structure.

LongTask is therefore best treated as one signal in user-experience troubleshooting. It can point to pressure in client-side rendering or state updates; correlated with action, resource, error, and business events on the session timeline, it helps engineers determine whether the bottleneck is on the client. If LongTask events cluster during a page refresh after the API has already returned, inspect list refreshes, layout calculations, and state-update frequency first.

When a Stutter Is Followed by an Error, Read It in Context

The stack trace may make many Flutter exceptions easy enough to understand. What it does not explain is why the exception occurred for this user, on this page, after this operation.

A state exception, for example, may come from:

The user taps submit 
-> the request goes out 
-> the server returns an unexpected structure 
-> Dart parsing logic enters the exception branch 
-> the Flutter state update fails 
-> an Error is reported

From the exception stack alone, it is hard to tell whether the parameter was invalid, the business API failed, a native capability failed, or the Flutter state machine failed to handle the response.

Flutter exception collection cannot rely on a single entry point. During initialization, the SDK hooks into several exception paths:

It catches unhandled exceptions inside the Zone through runZonedGuarded.
It takes over FlutterError.onError to handle synchronous Flutter framework exceptions.
It takes over PlatformDispatcher.instance.onError to handle uncaught exceptions at the platform dispatch layer.
It preserves the existing handler chain, minimizing interference with the application's current error-handling logic.
It provides onRUMErrorCallback so you can decide whether to keep reporting.
It uses setDumpError to control whether Flutter errors still go to the console.

For standard cases, just use start():

void main() {
  AlibabaCloudRUM().start(MyApp());
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await AlibabaCloudRUM().initialize();
  runApp(MyApp());
}

Note that if you choose initialize() and call runApp() yourself, some capabilities that depend on post-startup application state must be enabled separately at an appropriate time. For example, if you need LongTask detection, enable it after runApp() using the current public API:

await AlibabaCloudRUM().initLongTaskDetection();

For business troubleshooting, the exception alone is not enough. We recommend adding the following semantics:

Field	Meaning	Example
`business_stage`	Current business stage	`submit`, `pay`, `render_result`
`request_id`	Request ID	Correlates the request with server-side logs or traces
`error_stage`	Failure stage	`param`, `network`, `server`, `render`
`result_status`	End reason	`success`, `user_cancel`, `server_error`

When an error occurs, it is no longer just a Dart stack. Engineers can analyze it alongside the preceding action, resource, page state, and business stage.

Still Not Enough Clues: Add Context with Resource Snapshots While Respecting Data Boundaries

When investigating API problems, engineers usually want more context, such as error codes, request IDs, server response status, and whether required parameters are missing. But headers and payloads may contain user input, authentication data, or sensitive business fields, so they cannot be collected in full by default.

The SDK therefore uses an explicit Provider mechanism:

No header or payload is collected by default.
Collection happens only after you set a ResourceSnapshotProvider or Dio's onProvideSnapshots.
You are responsible for filtering, data masking, and ensuring compliance.
The SDK enforces size limits after the Dart-layer Provider or Dio callback returns: headers are measured by their JSON UTF-8 size and dropped if they exceed 64 KB; payloads are measured in UTF-8 bytes and truncated if they exceed 150 KB.

For critical requests, keep only the fields that help diagnosis and contain no sensitive content, for example:

ResourceSnapshots(
  requestHeaders: {
    'content-type': requestOptions.headers['content-type'] ?? '',
  },
  responsePayload: response?.data is Map
      ? {
          'code': response?.data['code'],
          'requestId': response?.data['requestId'],
        }.toString()
      : null,
);

Resource snapshots are meant to add troubleshooting context, but the size limits are not a substitute for your own data masking and compliance review. When integrating, prioritize the minimum diagnostic fields such as error codes and request identifiers, and avoid uploading user input or full business responses.

This is a deliberate design trade-off: the SDK provides a channel for collection and reporting, but it does not read application payloads on its own. This prevents monitoring logic from affecting the application's data flow or introducing compliance risks.

When Is an AI Chat Page Really Usable? Use View Metrics to Separate a Slow Page from a Slow Task

A Flutter page is usually not a static page that loads once. After a user opens an order, payment, content detail, or workbench page, the app may need to show the page container first, then load API data, render lists or rich text, and finally make key operations such as submit, refresh, and filter available. Page metrics are therefore better suited to answering a few business questions: did the page appear promptly, is the key content visible, are the main operations available, and can the user finish the current task quickly.

The Flutter RUM SDK collects standard routing scenarios through AlibabaCloudRUMNavigationObserver:

MaterialApp(
  navigatorObservers: [
    AlibabaCloudRUMNavigationObserver(
      ignoreRoutes: ['/splash'],
      enablePagePerf: true,
    ),
  ],
  home: HomePage(),
);

For non-standard page structures such as IndexedStack, PageView, and Tab containers, you can also use the manual API:

AlibabaCloudRUM().startView('OrderDetailPage');
AlibabaCloudRUM().stopView('OrderDetailPage');

Page performance collection is organized around a single Route lifecycle and focuses on the following metrics:

Metric	Meaning	How it is collected on the Flutter side	Questions it answers
TD	Transition Duration	Derived from TransitionRoute.animation state; in the current public implementation it is not necessarily reported as a separately queryable field	Whether entering a key page is slowed down by the page transition
FP	First Paint (FP)	Captured through WidgetsBinding.instance.addPostFrameCallback	Whether the page container appears quickly
FCP	First Contentful Paint (FCP)	Traverses RenderObject to detect content such as RenderImage, RenderParagraph, and TextureBox	When the first piece of business content, such as order, list, or detail content, appears
TTI	Time to Interactive (TTI)	Based on the effective element coverage rate; a fallback timing policy applies when the threshold is not reached	When the main operations, such as submit, refresh, and filter, become available

FP, FCP, and TTI in Flutter are estimates based on Flutter rendering and page structure, so they should not be interpreted in the same way as browser page metrics. For PlatformView, custom-painted widgets, or complex page containers, validate each metric against the page structure. Page performance data is reported through extension fields when the View event ends; the fields ultimately available for queries depend on the View extension map and each platform SDK's support.

For complex pages, read page metrics together with business stages. For example:

View: OrderDetailPage
-> FP / FCP / TTI
-> Action: Submit
-> Resource: /api/order/submit
-> Custom: business_stage = render_result
-> LongTask: page render

That lets you separate "the page itself opens slowly" from "the business processing is slow after the page opens."

After the Evidence Reaches RUM: Let STAROps Answer "Why Is the AI Still Waiting?"

Once RUM data reaches the platform, troubleshooting should not stop at handwritten queries. Start with the troubleshooting question, then let the observability platform help structure the analysis path.

Within the capabilities currently available in Cloud Monitor 2.0 (CMS 2.0), STAROps can serve as an assisted analysis entry point. CMS 2.0 refers here to the new generation of Cloud Monitor console capabilities, while STAROps provides assisted analysis of observability data. Product names, scope, and entry points are subject to current console availability.

For a Flutter app, the question is no longer "which table should I query?" It sounds much more like a real production troubleshooting question:

Which pages had abnormal wait times in the past hour? 
In sessions where nothing happened after the submit button was tapped, did LongTask events or slow requests appear at the same time? 
Are API errors concentrated in a specific version or a specific class of device? 
After a version upgrade, did Resource errors on the page and TTI increase at the same time?

STAROps helps by organizing these natural-language questions into an analysis path:

Start from the symptom. First describe the problem: "the page keeps spinning," "nothing happens on tap," "the API fails."

Narrow the impact. Use application version, operating system, device, page, region, and time window to define the affected scope.

Correlate session events. Link the action, page performance, resource, error, LongTask, and business events from the same user session.

Develop working hypotheses. Possible causes include client-side blocking, a slow API, retries along the path, rendering failures, or resource errors concentrated in one version.

Inspect individual samples. Return to a specific session, page path, request path, and exception context to test each hypothesis.

What STAROps can do, which entry points are available, and how well the analysis works all depend on what the console currently supports, and the results also depend on the completeness of the underlying RUM data. If pages are named inconsistently, Actions lack business semantics, resource snapshots are not masked and enriched as needed, or tracing is not connected end to end, assisted analysis can only see part of the picture. Automatic collection on the SDK side and semantic enrichment on the business side remain the foundation.

Bring the Troubleshooting Path into the SDK: Start with One Key Page

Trying to cover every Flutter page, network request, and business flow from day one increases integration costs and makes definitions harder to standardize. A better approach is to pick one representative critical page and establish a minimal end-to-end loop first.

1. Get RUM Running First

Standard case:

void main() {
  AlibabaCloudRUM().start(MyApp());
}

Custom startup flow:

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await AlibabaCloudRUM().initialize();
  runApp(MyApp());
}

2. Then Wire Up Pages, Network, and Behavior

Mark the key pages with AlibabaCloudRUMNavigationObserver or startView/stopView.

Collect the core business requests with AlibabaCloudRUMDioInterceptor or HttpOverrides.

Use AlibabaCloudActionCapture and ActionAnnotation to detect key operations such as submit, refresh, back, and retry.

3. Once the Basic Data Is Flowing, Add Business Semantics

Once view, action, resource, LongTask, and error collection are in place, RUM already records what the user did, whether requests succeeded, whether the page stuttered, and whether an exception occurred.

But these technical events do not fully explain the business process. Developers still need to know which flow the events belong to, which stage the user is waiting on, and how the operation ended. The application only needs to add enough context to answer those questions:

Context to add	Reference fields	Questions it answers
Flow correlation	flow_id, request_id	Whether these pages, operations, requests, and exceptions belong to the same flow
Execution stage	business_stage	Whether the wait occurs during the request, processing, parsing, or rendering stage
Execution result	result_status, error_stage	How the flow ended, and at which stage the exception occurred

In an AI conversation scenario, flow_id can map to one conversation or task, and business_stage can separate stages such as calling the model, streaming the response, and rendering the page. Design the field names and values around your actual business workflow; they are not built-in fields provided by the Flutter RUM SDK.

Design fields so that they explain problems; you do not need to cover every state change. Once these semantics are in place, the next step is to check that they show up in the same session as view, action, resource, LongTask, and error events.

4. Finally, Check That One Session Is Complete

After integration, do not just check whether individual metrics appear. Verify that RUM captures a complete user timeline:

View: OrderDetailPage
-> Action: Submit
-> Resource: /api/order/submit
-> Custom: business_stage / result_status
-> LongTask: page render
-> Error: optional

Once RUM can stitch this trace together, extend it step by step to more pages, more business processes, and more business semantics.

Back to the Beginning: Reconstruct the Wait, Don't Just Add More Logs

Production experience problems in a Flutter app usually cannot be explained by one API, one stack trace, or one tap. A report that "the page keeps spinning" may involve client-side interaction, a network request, server-side processing, page rendering, main Isolate blocking, and exception handling. Server-side logs alone cannot show the client-side page state or rendering behavior; a Flutter exception stack alone cannot show the preceding action and request path.

The Flutter RUM SDK brings these scattered events back into one user session: what the user did, whether the request went out, whether the API failed, whether the page stuttered, and whether an exception occurred on the same path. Instead of a pile of isolated logs, engineers get a coherent record they can trace, correlate, and review.

This approach is already implemented in the Alibaba Cloud RUM Flutter SDK. For the exact APIs, fields, and supported platforms, refer to the official release and the integration documentation. There is still room to go further in areas such as complex gesture detection, LongTask analysis, standardized business fields, compliance policies for resource snapshots, and field consistency across platforms.

For mobile developers, the goal of observability is not to collect more data. It is to break a real wait into a verifiable trace so engineers can understand the problem faster.

Learn more: https://www.alibabacloud.com/help/cms/cloudmonitor-2-0/access-to-mobile-applications-developed-through-flutter

From a Single Alert Card to One-Click RCA: How Tastien Built a Closed-Loop AIOps Practice Across 10,000+ Outlets

ObservabilityGuy — Fri, 28 Aug 2026 07:35:07 +0000

This article introduces how Tastien built a closed-loop AIOps system to unify alerts, enable AI-driven root cause analysis, and cut MTTR by 65% across its 10,000+ outlets.

Founded in 2012, Fuzhou Tastien Catering Management Co., Ltd. is a restaurant chain built around the "hand-rolled, freshly baked" Chinese burger. Drawing on traditional Chinese pastry craft, Tastien developed a distinctively textured hand-rolled, freshly baked burger bun and created an entirely new category: the Chinese burger. Strong R&D, consistently high product quality, and a convenient, attentive service experience have made Tastien one of China's leading restaurant brands. Today it operates more than 10,000 outlets across over 300 cities, with a large and loyal consumer following.

At a chain of this size, operational systems span dozens of service chains — outlet POS, supply chain, Customer Relations Management, online ordering, and more. In the AI era, as the business grows and iterates at speed, monitoring covers ever more ground and teams configure ever more monitoring and alerting — which brings problems of its own. Alerts arrive from many sources: CloudMonitor, ARMS, Simple Log Service, custom business alerts, and certificate alerts. Because that alert data sits in separate systems with different data formats, notification methods, and handling flows, the cost of configuring alerts, delivering notifications, responding and investigating, capturing knowledge, and closing the loop keeps climbing.

The Business Challenge: What Happens After the Alert Determines O&M Efficiency

In a multi-business cloud environment, companies rarely lack monitoring tools. What they lack is the ability to turn an alert into a management event that teams can work on together, track, and reuse. One fault can fire alert after alert until the important details are buried in the flood. On-call engineers switch between platforms and stitch the context together from experience. AI returns a diagnosis, but with no human feedback, no one can tell which analyses were right and which need work.

Traditional monitoring solved the problem of detecting faults. Alert management, and the collaboration that follows an alert, still runs into the same recurring problems:

Alerts scattered across sources: Alerts come from many places, and the data lives in four or five different systems. Each monitoring system needs its own contacts, notification methods, and delivery policies — expensive to maintain and easy to get wrong. Worse, alert content and status are expressed differently in each system, so on-call engineers struggle to read the alert status quickly, locate the problem, or notify the right owner in time.
Heavy duplicate noise: A single alert rule usually covers many instances, sometimes dozens. When one fault trips them all at once, a flood of duplicate Lark messages and phone calls follows within minutes — an alert storm. Duplicates add to the judgment and handling load on on-call engineers and drown out what actually matters, so critical alerts get missed.
Diagnosis that depends on individual experience: When an alert fires, the root cause may lie in a metric, a Simple Log Service log, an application trace, or an operational system. Investigating means switching back and forth across platforms and manually correlating monitoring, application, container, and log context — slow work that rests heavily on how much the on-call engineer happens to know.
No closed loop, no accumulated knowledge: Alert recovery is not the same as closing the loop. Without structured capture of root causes and solutions, past experience is hard to retrieve and reuse, and it never feeds back into the next diagnosis.

In our view, what happens after the alert is what decides O&M efficiency. Solving these problems calls for a single intelligent AIOps platform that ingests alerts from every source, standardizes them, and orchestrates them — delivering each alert to the right O&M engineer, helping them locate and fix problems fast, limiting the impact, and capturing what was learned so the alert loop keeps improving. The goal was never to add one more alert page, but to give every alert a complete lifecycle.

The Solution: One Alert Entry Point, an AI Feedback Loop, and a Digital Worker Matrix

To tackle scattered sources, duplicate noise, slow root cause work, and lost knowledge, Tastien built on Alibaba Cloud CloudMonitor 2.0 (CMS 2.0) and the full-stack AIOps platform STAROps. The Unified Alert Center in CMS 2.0 became the single entry point, a low-code workflow was added to the existing O&M platform as the alert orchestration layer, and together they close the loop:

CMS 2.0 multi-source alert ingestion → low-code workflow orchestration → automatic convergence → Lark collaboration → STAROps AI-assisted Analysis → root cause information added → AI rated → knowledge captured

(1) Unified Alert Management in CloudMonitor 2.0 + Workflow Orchestration: One Entry Point for Every Alert Source via Event Integration and Subscription

Tastien routes ARMS, CMS 1.0, Simple Log Service alerts, Prometheus, and custom alerts into the Event Center through the event integration capability of CMS 2.0. A workflow then orchestrates the rules that normalize them into management events:

Webhook ingestion → source detection → field standardization → convergence decision → group routing and on-call scheduling → card generation or update → write to data store → tiered escalation and feedback.

The workflow picks a parse branch by alert source and transforms the title, level, resource, status, and other information into one common context. It then invokes alert convergence and group routing to decide whether to send a new card, update an existing one, or close it out on recovery, and it applies the matching notification and escalation policy for P1, P2, or P3. Claims, status updates, and root cause feedback raised on the Lark card flow onward through the same orchestration chain. This keeps the visual orchestration flexible while keeping long-lived state, consistency, and historical facts out of the workflow. Adding a new alert source usually means adding a parse branch and reusing the shared downstream stages, with no need to rebuild the whole handling chain.

(2) Lark Alert Cards + AI Root Cause Analysis: Claim, Ask, and Track End-to-End on One Card

The platform keeps real-time status, the collaboration entry point, and historical facts apart: real-time status drives alert convergence and card updates, Lark carries the live collaboration, and historical data preserves confirmed root causes, feedback, and handling records.
On-call engineers claim an alert, view its status, and reference the card to ask the AI what caused it — all from the card itself. The system pulls in the current alert context automatically, so no one has to restate the situation or search across platforms.
Typical scenarios delivered on the Lark alert card:
Ask about O&M alerts: describe the query in natural language, such as today's alert volume, unclaimed alerts, or top alert rules
Create Alert Rules: create cloud alerts and custom alert rules in natural language
Ask follow-up questions from an alert card: reference the card, skip the recap, and let the AI pull in the context
One-click root cause analysis: reference the card to ask the AI, which queries historical root causes and invokes STAROps root cause analysis to reach a conclusion, streaming the diagnosis in real time so the process stays transparent
Human-AI collaboration: interactive execution, with manual review for high-privilege commands

This turns the alert card from a notification into a handling entry point: who owns the problem, how far the analysis has progressed, and what the root cause turned out to be all accumulate around the same management event.

(3) The AI Feedback Loop: Every Diagnosis Captured as a Triplet of Context, Human-Confirmed Root Cause, and Rating

Whether AI root cause analysis is reliable cannot be settled by impressions on the product side; it takes data. Tastien embedded a minimal but complete feedback mechanism into the handling chain:

Step 1 · AI produces a diagnosis: after an on-call engineer references the alert card and asks a question, the AI produces a diagnosis based on the injected management event context and the historical root cause repository for that service.

Step 2 · The engineer fills in the actual root cause: once the management event has recovered (or once the root cause is confirmed during handling), the on-call engineer describes the actual root cause through the "Fill in Root Cause" entry on the card — free text, but it must state clearly what actually happened.

Step 3 · Rate AI accuracy: after the root cause is filled in, the system opens a rating entry with only two options — "AI is accurate" or "AI is inaccurate". If the engineer selects "AI is inaccurate", they can add an optional one-line note (for example, "AI attributed the root cause downstream; it was actually an upstream traffic problem").

As these records build up, they deliver two direct benefits:

A historical root cause repository: the next time a similar alert hits the same service, the AI reads that service's historical root cause records first in its Step 1 diagnosis, drawing on real past root causes rather than reasoning from current metrics alone. The more it is used, the closer its conclusions track the customer's actual business.
Diagnosis accuracy statistics and scenario-based improvement: the team can measure AI accuracy by service and alert_type, then target improvements at specific scenarios.

(4) The STAROps Digital Worker Matrix: Skill Encapsulation, Mission Orchestration, and One-Click RCA

On top of the alert loop, Tastien built a second layer of capability with STAROps: encoding the troubleshooting steps held in senior SREs' heads as Agent capabilities the platform can schedule.

Skill Encapsulation — Turning Troubleshooting SOPs into Reusable Capability Units

Each Skill defines the complete troubleshooting path for one specific fault scenario: trigger conditions (which alert or metric combination should invoke the Skill), data collection steps (which metrics, logs, and traces to query, and in what order), judgment logic (which metric combination points to which root cause), and output format (a structured diagnosis plus recommended actions).

Mission Orchestration — Running Skills in Parallel to Form an Inspection Matrix

A single Skill handles diagnosis for a single scenario, but daily inspection needs dozens of Skills running in parallel under line-of-business and time-window policies. Using the Mission (long-running task) capability in STAROps, Tastien orchestrates multiple inspection Skills into continuously running digital workers. After each inspection round, the Agent outputs a structured inspection report: normal items collapsed, abnormal items highlighted with a diagnosis and recommended actions. The SRE's day shifts from "watching monitoring dashboards in shifts" to "reading the inspection report each morning and handling the items flagged in red".

Taken together, what Tastien built is not one more alert page. It turns alerts from messages into management events, AI output into data that can be rated, and a one-off incident response into a reusable O&M asset.

Only when every alert is grouped correctly, handled promptly, reviewed in full, and fed back into the next diagnosis does an O&M platform move from a notification tool to an engineering system that keeps learning.

Results: From an Engineering Loop to Measurable Gains

Significant reduction in alert noise: Through unified alert convergence and AI-based noise reduction in CloudMonitor 2.0, the alert compression rate reached 95%, and the invalid alert rate decreased by 15%.
Substantial improvement in incident resolution efficiency: With AI-assisted root cause analysis and end-to-end collaboration through Lark, the Mean Time to Recovery (MTTR) was reduced by 65%.
Effective accumulation of O&M expertise: A historical root cause library covering more than 40 business-specific scenarios has been established, enabling sustainable reuse of operational experience and continuous evolution of AI models.