DEV Community: ObservabilityGuy

Alibaba Cloud Partners with Datadog Close Go's Last Observability Gap

ObservabilityGuy — Wed, 09 Sep 2026 02:13:59 +0000

This article introduces OpenTelemetry Go Compile-Time Instrumentation v1, enabling zero-code, build-time observability for Go applications.

Go was the last mainstream language without zero-code observability. In July 2026, the OpenTelemetry Go Compile-Time Instrumentation project shipped its stable v1 release. Launched jointly by Alibaba and Datadog and developed through a year and a half of community collaboration, it lets Go developers add distributed tracing and metrics collection to an application with a single command, without touching any business code.

This article walks through how this milestone works, how to use it, and when to choose it.

1. Why Go Has Long Lacked Zero-Code Observability

Java has -javaagent, Python has sitecustomize, Node.js has --require.NET has CLR Profiler. AI Agents in these languages inject agents dynamically at runtime, so developers get traces and metrics without changing a line of code.

Go can't do that, and the reason is fundamental: Go compiles to a static binary — no VM, no bytecode, no class-loading hooks. Once go build finishes, you have a self-contained machine-code file with nowhere to "attach" an agent at runtime.

That has long left Go developers with two options:

Manual instrumentation: call span.Start() / span.End() explicitly in every HTTP handler and around every DB call — intrusive, and easy to miss spots
eBPF observation from the outside: capture network calls in the kernel — zero-touch, but you see only L4/L7 protocol information, with no business semantics

A wide gap separates the two. For a platform engineering team running hundreds of Go microservices, "add tracing by hand to every service" isn't realistic. eBPF covers a lot of ground but can't reach inside the code — if you want to know which SQL query was slowest in a request, eBPF can only tell you "this TCP connection took 200 ms"; it gives you nothing at the database/SQL level.

Compile-time instrumentation fills exactly that gap: probes are injected during go build, and the resulting binary carries observability with it. No code changes, no external AI Agent.

2. Compile-Time Instrumentation: Injecting Agents at Build Time with -toolexec

The Go toolchain has a little-known but very powerful extension point: -toolexec.

When you run go build, the go command is really orchestrating lower-level tools such as compile and link. -toolexec lets you name a wrapper program, and the Go toolchain routes every invocation of compile/link through that wrapper first — much the way Unix strace or time wraps a command.

OpenTelemetry Go Compile-Time Instrumentation builds on this mechanism. Its core tool, otelc, acts as a wrapper around -toolexec. As the compiler processes each package's source, otelc does the following:

Parse the AST: read the current package's abstract syntax tree
Match rules: check whether a registered instrumentation rule applies (for example, "this is ListenAndServe in net/http")
Inject code: weave tracing/metrics code into the target function before compilation
Pass through to the compiler: hand the rewritten source to the original compile for normal compilation

The binary you end up with has the agent code built in. At runtime there is no extra agent process, no sidecar, and no eBPF program to load.

How this fundamentally differs from a Java agent: Java injects logic at runtime by rewriting bytecode, which incurs runtime overhead (every class load passes through a transformer). Go compile-time instrumentation does all the rewriting during the build, so there is no additional runtime overhead — what you get is an ordinary Go binary that happens to contain tracing code.

This is also what makes the approach so CI/CD-friendly: in your build pipeline you replace go build with otelc go build. No changes to your deployment architecture, no Pod spec edits, no DaemonSet.

3. What v1 Supports: Coverage from net/http to gRPC

As the first stable release, v1 focuses on the most heavily used library categories in the Go ecosystem:

Library	Type	Signals produced
net/http	HTTP client/server	Trace spans + HTTP semantic attributes (method, status_code, route)
Database/SQL	Database access	DB spans + statement summary + connection information
google.golang.org/grpc	RPC framework	Client/Server spans + gRPC semantic attributes
github.com/redis/go-redis	Redis client	Redis command spans
Go runtime	Runtime metrics	GC, goroutine, memory metrics

A few key design decisions:

Rule-based architecture. The instrumentation logic for each library is defined as a rule, and a rule describes the package paths and function signatures to match plus the code template to inject. This means the community can contribute new rules independently, without touching the core framework. After v1, supporting a new library essentially means submitting a new rule.

Semantic Conventions compliance. Every span and metric produced follows the OpenTelemetry Semantic Conventions — attribute names, span names, and metric units are fully standardized. Whatever backend you use (Jaeger, Tempo, Simple Log Service, ARMS), the data means the same thing.

Automatic discovery. By default, otelc scans your go.mod dependency tree and automatically finds and enables every library covered by a registered rule. You don't have to declare "instrument net/http" — if you use it, it gets instrumented.

v1 deliberately prioritized a focused, high-quality core over breadth. Every supported library has been through full correctness tests and performance benchmarks. Later releases will keep expanding coverage.

4. Get Started in 3 Minutes: Add Tracing with One Command

Install otelc:

go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest

Option 1: replace the build command directly

otelc go build -o myapp .

That's it. The resulting myapp already has tracing code built in. Set OTEL_EXPORTER_OTLP_ENDPOINT at startup and trace data will flow to your Collector automatically.

Option 2: leave the build command unchanged (CI/CD friendly)

otelc setup 
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'" 
go build -o myapp .

This fits better when you already have a complex Makefile or CI pipeline — you add two lines of setup to the build environment and leave the existing go build command alone.

Dockerfile integration example:

FROM golang:1.23 AS builder
RUN go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest
WORKDIR /app
COPY . .
RUN otelc go build -o /myapp .
FROM gcr.io/distroless/base
COPY --from=builder /myapp /myapp
ENTRYPOINT ["/myapp"]

Image size is unaffected: otelc is only used during the build, and the final image contains only the compiled output.

5. Choosing Among Three Paths: Compile-Time vs. eBPF vs. Manual

Go observability now has three complementary paths, not competing ones:

Dimension	Compile-time instrumentation	OpenTelemetry eBPF Instrumentation (OBI)	Manual instrumentation (Go API)
Prerequisites	Able to recompile the source	Linux kernel ≥ 4.x, privileges	Able to modify the source
Code intrusion	None	None	High
Runtime overhead	Very low (code is inlined)	Low (collected in kernel space)	Depends on the implementation
Depth of coverage	Function level (including third-party dependencies)	Protocol level (HTTP/gRPC/SQL)	Any granularity
Multi-language support	Go only	Go/Java/Python/Node.js, etc.	Go only
Deployment changes	Change the build command	Deploy a DaemonSet	Change code + redeploy
Best fit	Platform engineering teams rolling out observability uniformly	Existing services, multi-language clusters	Business logic that needs custom spans

How to decide:

If you can rebuild your services and want zero-code coverage with low overhead → compile-time instrumentation
If you have many existing services, don't want to recompile them one by one, or run a mixed-language cluster →OBI
If you need custom spans inside specific business logic (business semantics such as "user places order") → manual instrumentation

You can combine all three. Compile-time instrumentation covers the generic spans from the standard library and third-party dependencies, manual instrumentation adds business-semantic spans, and the two sets of spans stitch into one trace automatically. eBPF then serves as the fallback for older services you can't recompile yet.

In practice, for a typical Go microservice cluster the most pragmatic strategy is this: use compile-time instrumentation plus a little manual instrumentation for new services; cover existing services with eBPF first, then move them to compile-time instrumentation in CI over time.

6. Community Collaboration and Next Steps

How this project came about is itself an interesting case of open-source collaboration.

In early 2025, Alibaba and Datadog were each exploring Go compile-time instrumentation internally and discovered each other's work. Rather than build separately and then fight over the standard in the community, the two companies merged their efforts into the OpenTelemetry community and created a dedicated Special Interest Group (SIG) under CNCF to drive the work in a vendor-neutral way.

Over a year and a half, the project went all the way from PoC to stable. Key milestones:

SIG formed (2025 Q1): technical direction and governance structure settled

Core framework landed (2025 H1): rule engine, AST rewriting, test infra

Community growth (2025 H2): new contributors joined through CNCF LFX Mentorship — among them Azhar Momin, who went from mentee to approver

v1 released (2026 Q3): first stable version, covering 5 core library categories

What's on the roadmap next:

More instrumentation rules: Kafka, MongoDB, AWS SDK, and others
Registry integration: discover and install community-contributed rules through the OpenTelemetry Registry
Build performance optimization: reduce otelc's impact on compile time
Validation in more scenarios: production case studies and performance benchmarks

7.Final Thoughts

Go's observability gap is finally closed.

If you're a platform engineer running observability infrastructure for dozens or hundreds of Go services, otelc go build may be the highest-ROI change you can make: one command, full coverage, nothing intrusive.

If you maintain a library or are simply interested in OpenTelemetry, we'd welcome your rule contributions. Writing an instrumentation rule for a library is far easier than implementing an SDK wrapper from scratch — a rule is essentially a declarative description of what code to inject, in which function, at which position.

Resources:

Project repository: github.com/open-telemetry/opentelemetry-go-compile-instrumentation
Quick-start docs: Getting Started in the project README
CNCF Slack channel: #otel-go-compile-instrumentation
OBI: github.com/open-telemetry/opentelemetry-go-instrumentation

STAROps Synthetic Monitoring Intelligent Analysis in Practice: From Task Analysis to Cross-Domain APM Fault Demarcation

ObservabilityGuy — Fri, 04 Sep 2026 05:46:11 +0000

This article explains how STAROps integrates Synthetic Monitoring, SLS, UModel, and APM to automatically analyze external access anomalies and accurately pinpoint cross-domain faults.

When a user reports that "the site won't load" or "the API suddenly got slow", network monitoring usually flags the anomaly quickly. The hard part comes next: did the fault occur on the network path between the user and the server, or inside the server-side application? Does it hit one region or ISP only, or every user? Should you call the network team, the application team, or the owner of a downstream dependency?

Synthetic Monitoring is the tool that watches this external access experience. Probe nodes spread across regions and ISPs follow fixed rules to simulate real users visiting a website, an API, or a domain name, and they record data for each stage: DNS resolution, TCP connection, TLS handshake, and server response. Traditional synthetic monitoring reports, however, are better at showing what happened than at explaining why it happened.

STAROps brings intelligent O&M analysis to observability data. Its Agent automatically queries task configuration, synthetic monitoring logs, and application call chains, then links evidence that used to sit in separate systems. In the end it answers four questions: where the anomaly occurred, which user paths it affected, which side owns the problem, and what data backs the conclusion.

This article is for product managers, developers, and O&M engineers who are new to Synthetic Monitoring intelligent analysis. It explains why STAROps connects Synthetic Monitoring, SLS, UModel, and APM, and how that chain moves you from spotting an anomaly to explaining it and pointing at a fix. By the end, you will understand where traditional synthetic monitoring analysis breaks down, how STAROps assembles evidence across systems, and how a network problem is told apart from a backend one.

Getting Started: Key Concepts in the Analysis Chain

Synthetic Monitoring: Probe nodes actively visit a target website, API, or domain name and continuously check availability and performance from the user's point of view.
STAROps Agent: The intelligent analysis unit that queries data, aggregates evidence, judges faults, and generates reports. Referred to as the Agent below.
SLS: Alibaba Cloud Simple Log Service, which stores the raw execution record of every synthetic check. Think of it as the log warehouse you search for factual evidence during analysis.
UModel: A unified model that describes observable objects such as tasks and applications, plus the relationships between them. Think of it as a map that tells the Agent who each object is, where its data lives, and how the objects connect.
APM: Application Performance Monitoring, used to watch how a request executes after it reaches the backend. One complete request is a Trace, and each processing step within it is a Span.
Workspace: An isolated scope of observability data and resources. The Agent must pin down the Workspace before it can locate the right task, logs, and APM data.

1. Background: Gaps in Traditional Synthetic Monitoring Analysis

Figure 1: Main gaps in traditional synthetic monitoring analysis

Traditional analysis usually starts with a synthetic monitoring report: check availability rate, response time, and error codes, then switch by hand between logs, task configuration, and APM pages to look for a cause. These metrics work as a health overview, but they rarely add up to verifiable root cause evidence, so four gaps remain in troubleshooting.

First, task context is missing. The same error code can mean completely different things under different task types, target addresses, DNS configurations, assertion rules, and monitoring point coverage. Repeating the error code alone tells you nothing about whether the cause is a misconfiguration, a broken target service, or a failure at one protocol stage.

Second, raw evidence is missing. OpenAPI is the standard query interface the system exposes, and central-side metrics are already aggregated statistics; both suit stable queries and health overviews, but neither carries all the detail an ad-hoc drill-down needs. Stage durations, city and ISP, target IP, CNAME, raw DNS answers, response headers, assertion results, and Trace context usually live only in the raw synthetic monitoring records in the user's SLS.

Third, protocol stages are not opened up. A single HTTP check may run through DNS, TCP, TLS, request send, first-packet wait, HTTP response, response download, and content assertion. Look only at total duration and the final error code, and problems from different layers blur together.

Fourth, synthetic monitoring and the backend application are not linked. Synthetic Monitoring can tell you that external access failed or slowed down, but once the request enters the backend, the problem may sit in business logic, the database, the cache, or a downstream dependency. In APM, the server span marks where the backend receives and handles the request, and child Spans record internal steps such as database, cache, and downstream calls. Without that evidence, the Agent cannot reliably pinpoint a problem inside the backend.

Synthetic Monitoring intelligent analysis therefore needs a complete chain: task identification → data location → raw sample query → field filtering and combined analysis → protocol-stage drill-down → link diagram rendering → cross-domain APM alignment → final fault demarcation.

2. Core Capabilities

Figure 2: The five core capabilities of STAROps Synthetic Monitoring intelligent analysis

1. Synthetic Monitoring Task Analysis

Every synthetic monitoring task carries a unique taskId. Once you supply a taskId, a task name, or a target address, the Agent locates the matching synthetics.task entity — the structured description UModel keeps for that task. It then reads the task type, target, probe frequency, monitoring points, timeout, assertions, and Trace configuration, and queries raw synthetic monitoring samples within the specified time range.

The analysis does more than compute sample count, success rate, availability rate, response time, and error distribution. It also filters, aggregates, and compares by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Even when a task looks stable overall, single-task analysis still checks protocol stages, localized slow points, and configuration risks instead of simply reporting "running normally".

Coverage today includes HTTP/HTTPS, Ping, TCP, UDP, DNS, DNSTRACE, SMTP, POP3, FTP, Traceroute, MTR, API, Multi, Browser, WebSocket, and SSL. Each task type follows its own analysis path: Ping focuses on packet loss and RTT, API and Multi on failed requests and steps, Browser on the main document, page timing, and the resource waterfall.

RTT (round-trip time) is the time data takes to travel from the probe to the target and back. API, Multi, and Browser correspond to API requests, multi-step tasks, and browser page loads respectively.

2. Intelligent Workspace Inspection

Single-task analysis answers "why is this task failing"; Workspace inspection answers "which tasks in this group deserve attention first". A Workspace is an isolated observability space that holds the tasks, logs, and related resources belonging to one user or business scope.

Workspace inspection first rolls up all synthetic monitoring tasks into an overall health view, then drills down into the tasks that matter. Task selection weighs status severity, failure count, availability rate, latency, error concentration, failing-stage concentration, and how long the anomaly has lasted, so the ranking never rests on a single metric.

Inspection supports three cycle scenarios:

High-frequency monitoring: looks at the last 30 minutes for wide-scale unavailability, clustered timeouts, and high-priority risks, and analyzes the Top 1–3 tasks in depth.
Daily diagnosis: analyzes the last 24 hours against the same window the previous day, and reports trends, lingering risks, and priority tasks.
Weekly review: analyzes the last 7 days against the previous week, and identifies persistent anomalies, repeated fluctuations, recovered risks, and administration items.

3. Fine-Grained Protocol Diagnosis

The Agent picks protocol evidence to match the task type rather than explaining every task with one set of metrics:

HTTP/HTTPS: analyzes stage by stage along DNS → TCP → TLS → request send → first-packet wait → HTTP response → response download → assertion.
DNS: checks the query object, DNS Server, RCODE, CNAME, resolved IP, SOA, expected match, and whether the domain name exists.
WebSocket: additionally checks whether the HTTP Upgrade completed.
API and Multi: pinpoint the failing request or step.
Browser: separates main-document, resource-loading, page-performance, and operation-step anomalies.

If network protocols are new to you: DNS translates a domain name into an IP address, TCP establishes a reliable connection, and TLS encrypts the traffic and validates the certificate. RCODE is the result status returned by the DNS server, CNAME marks a domain alias, and SOA records the basic information of the domain's authoritative zone.

Diagnosis covers configuration problems as well as data problems. A DNS query object mistakenly set to a full URL, an HTTP 200 whose content assertion fails, a disabled task, a query window shorter than the probe interval, or no matching samples in SLS — each is flagged on its own and never lumped in as a target service anomaly.

4. Request Link Visualization

The synthetic monitoring access link diagram is generated dynamically from real samples. The first node is always the probe, and only the protocol stages the request actually reached appear after it. If the request fails at DNS, the diagram stops at DNS and does not draw TCP, TLS, or the backend service.

The diagram follows a "stage node → anomaly node" structure. A DNS anomaly, for example, renders as "probe → DNS resolution → resolution anomaly (NXDOMAIN)" rather than jumping straight from the probe to the anomaly. Normal nodes, affected paths, and the final root cause use different styles, so an anomaly propagation path is not mistaken for the root cause.

Figure 3: Synthetic monitoring access link diagram generated from the actual execution stages in STAROps task analysis

The diagram does not depend on APM. With no Trace available, it still shows the synthetic monitoring access path; when a backend Trace exists, an APM call chain diagram is added to show the backend entry point, API, business processing, downstream dependencies, and the specific point of failure. The trace_id ties the two diagrams to the same request.

5. Cross-Domain APM Fault Demarcation

When an HTTP, API, Multi, or Browser task carries Trace evidence such as trace_id, traceparent, or traceInfo, the Agent picks a failed or high-latency sample and queries the APM Trace itself, instead of merely telling you to look it up.

A Trace is the full path of a request from the probe into the backend and on to other components; a Span is one step along that path. Linked analysis aligns four kinds of Span:

Synthetic monitoring client span: the external view, from the probe sending the request to receiving the response.
Backend server span: the entry view, from the request reaching the application to the application returning a response.
Local business Span: the execution of a Handler, business method, or internal computation.
Downstream dependency Span: a call to a database, cache, RPC, or external HTTP service.

By comparing synthetic monitoring stage durations, the server span, child Spans, and the first concrete error, the Agent can place the fault on the network path, the path before the application entry, the backend service, or a downstream dependency — or state that evidence is insufficient. A 502 from a backend API may be only the surface symptom; if the first concrete error appears in a downstream DNS, database, or external service Span, the conclusion moves to the downstream dependency.

3. Synthetic Monitoring Architecture and Technical Approach

Figure 4: Analysis architecture combining UModel, SLS, and APM

Data Organization: UModel and SLS Working in Two Layers

Think of UModel as the map and SLS as the field notes. UModel holds relatively stable object information: which task this is, what it targets, which Workspace it belongs to, where its data lives, and how it relates to other objects. SLS holds the execution facts that keep arriving: which probe hit which target at what time, which protocol stages it went through, and whether it succeeded.

Creating a synthetic monitoring task produces a synthetics.task entity in UModel that holds metadata such as taskId, task type, target address, probe configuration, Workspace, SLS data coordinates, and Trace configuration. Every real probe run then writes one or more execution records into SLS. One task entity therefore maps to many synthetic monitoring samples; each probe run does not create a new entity.

When a request arrives, the Agent first queries this map through UModelSearch to locate the task entity and its log location, then enters the matching SLS Project and Logstore to query samples by taskId and time range. A Project is the isolated resource space in SLS, and a Logstore is the data container inside it that holds one class of logs.

Analysis Strategy: Overall Aggregation, Risk Screening, and Protocol Drill-Down

With the samples in hand, the Agent first sizes up the scope and distribution of the anomaly, then drills down step by step into specific protocol stages. Single-task analysis aggregates by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Workspace inspection instead rolls up all tasks first, then screens out the ones with low availability rate, high latency, concentrated errors, or persistent anomalies.

After screening, the Agent reads representative failed or high-latency samples and takes a different analysis path per task type. An HTTP task, for instance, is checked in order for DNS, TCP, TLS, request send, first-packet wait, response download, HTTP status code, and content assertion; a DNS task goes on to read RCODE, the number of answer records, CNAME, resolved IP, SOA, DNS Server, and the match result.

This "overall aggregation → risk screening → protocol drill-down → representative sample" approach separates resolution failures, connection problems, certificate issues, slow server-side processing, and content assertion failures, instead of explaining only the final error code.

Cross-Domain Linkage: From the Synthetic Monitoring Domain into the APM Domain

The analysis chain can continue into APM only when the synthetic monitoring sample carries Trace information. For any task with Trace enabled, the system generates a synthetic APM service and uses the same_as relationship in UModel to map synthetics.task to that service permanently. For the Agent, this relationship is the signpost from the synthetic monitoring domain into the APM domain.

same_as answers "which synthetic APM object corresponds to this synthetic monitoring task"; trace_id answers "which applications and call nodes did this particular request pass through inside the backend". The first is a stable task-level relationship, the second a precise per-request association. Note one thing in particular: same_as does not equate the synthetic monitoring task with the real backend application.

Inside APM, the Agent looks at the synthetic monitoring client span, the backend server span, local business Spans, and downstream dependency Spans together:

If synthetic monitoring duration rises and the server span rises with it, the problem leans toward the backend service.
If most of the time sits in a database or downstream call, you can narrow it to that dependency.
If the client span is slow while the server span is short, the problem more likely lies in the network, CDN, gateway, queuing, or another uninstrumented segment.
If DNS, TCP, or TLS already failed and no server span exists, the request never reached the application.

If a sample has a trace_id but complete backend Spans are missing, the Agent explicitly marks "insufficient APM evidence" rather than reading "no data found" as "backend is normal".

Design Principle

The STAROps Agent does not hand a fixed report to a large model for summarizing. It builds a query plan first, then constrains its conclusions with real data. UModel serves as the semantic control plane that identifies objects and relationships, and the user-side Logstore serves as the data plane that holds observed facts: indexed fields screen samples fast, and SPL queries then fetch and process finer protocol details. Here, an index is a pre-built field for fast retrieval, and SPL is the query language used to filter, aggregate, and transform log data.

Different tasks load different evidence models, normalizing heterogeneous logs into five layers: task, single execution, protocol stage, blast radius, and call chain. When a sample carries Trace information, the Agent follows trace_id into APM and aligns the external client span with the backend server span, business Spans, and downstream dependency Spans. The output is not an empirical guess a large model made from field names; it is a traceable fault demarcation conclusion constrained by raw samples, aggregated results, protocol semantics, and cross-domain Traces.

End-to-End Analysis Chain

Parse the user input and decide whether the scope is a single task or a Workspace.
Query synthetics.task to obtain the task configuration and SLS data coordinates.
Fix the time window and calibrate time units.
Aggregate overall results and screen for risk using indexed fields.
For abnormal tasks, read representative samples and unindexed protocol fields.
Enter the dedicated analysis for DNS, HTTP, Browser, API, Multi, or another type as appropriate.
If a Trace exists, pick a failed or high-latency sample and query the APM call chain.
Align task configuration, synthetic monitoring stages, blast radius, and Span evidence, then output the fault demarcation conclusion, link diagram, recommended actions, and data limitations.

The core strategy is "overall aggregation → risk screening → protocol drill-down → representative sample → cross-domain alignment". It keeps query volume under control across long time windows and large Workspaces while preserving the raw evidence needed to pinpoint the failing stage and the root cause.

4. How to Use

For task analysis, simply supply a taskId, a task name, or a Workspace, for example:

Analyze the synthetic monitoring tasks over the last 30 minutes.

You do not have to write log queries yourself. The Agent turns a natural-language question into concrete steps: task location, sample query, protocol analysis, and APM linkage.

The Agent then completes the following steps automatically:

Identify the scope and confirm whether this is single-task analysis or Workspace inspection.
Find synthetics.task through UModelSearch or the task query capability.
Locate the user-side SLS Project and Logstore.
Query raw synthetic monitoring samples within the fixed time window.
Run protocol-stage analysis according to the task type.
If Trace evidence exists, go on to query the APM Trace.
Output the complete Synthetic Monitoring intelligent analysis report.

For ongoing inspection, choose the high-frequency monitoring, daily diagnosis, or weekly review template. The inspection scans the synthetic monitoring tasks in the Workspace on a schedule, screens out risky tasks, and generates a report automatically.

A "Long-Running Task" in STAROps is an analysis task that repeats automatically on a schedule; a "Digital Employee" is an Agent pre-configured with skills, tools, and data permissions. Combine the two and synthetic monitoring inspection runs on schedule and pushes its results to the people you designate.

Synthetic Monitoring Task Analysis

Figure 5: Starting a synthetic monitoring task analysis in the STAROps console

In the STAROps console, you can ask about synthetic monitoring task details directly, for example "Analyze the synthetic monitoring tasks over the last half hour", "Analyze why a given synthetic monitoring task is failing", or "Show the synthetic monitoring tasks whose availability rate dropped sharply in the last day".

Synthetic Monitoring Inspection Tasks

Figure 6: Opening Long-Running Task in STAROps

Click "Long-Running Task" in the upper-left corner of the STAROps console.

Figure 7: Creating a synthetic monitoring inspection task and configuring the Workspace

Click "Create Task" in the upper-right corner of the page, select a Digital Employee and the Workspace the task belongs to, and enter a prompt such as "synthetic monitoring inspection".
Select one of the three synthetic monitoring inspection templates or combine them, and describe the inspection details in text; by default the inspection runs at Workspace level.

Figure 8: Configuring notification recipients and adding the job to Long-Running Task

4. Click "Configure Now" to add notification recipients, then click "Add to Long-Running Task" once you have made your selection.

5. Confirm the configuration is correct, then click "Confirm Execution".

Figure 9: Viewing the inspection report generated by the Long-Running Task

6. Click "Report" at the top of the page to view the detailed inspection results.

5. Fault Demarcation Logic

1. Locate the Layer Where the Request Stopped

An HTTP call chain runs DNS → TCP → TLS → request send → first-packet wait → HTTP response → download → assertion. The deepest stage the Agent can observe tells it whether the request reached the service entry point.

A DNS failure with no TCP evidence, for example, means the request never established a connection. An HTTP 502 shows that DNS resolution, the TCP handshake, and the response header return all completed, so the problem cannot be a first-packet wait timeout. An HTTP 200 with a failed assertion means the service is reachable and the failure sits in the response content or the business logic.

2. Determine the Scope of Impact and Temporal Pattern

Simultaneous failures across multiple cities and ISPs point to the target service, a public entry point, or a cross-network path. Concentrated failures in a single city or ISP suggest a regional network or ISP link issue. If only some monitoring points fail within the same city and ISP, the Agent drills down by client_id, DNS Server, local DNS, and egress path when these fields are available.

The Agent also separates continuous, sudden, intermittent, and recovered anomalies over time. It never generalizes a single spike into a continuous failure, and averages never stand in for P95, maximum values, or anomalous time buckets.

3. Prioritize DNS Evidence Over Outer Error Codes

The DNS error_code reflects the probe's local judgment, while RCODE is the protocol-layer response from the DNS server — the two are not interchangeable.

When raw_exception is present, the Agent reads rcode, rcode_name, and the answer record count first, to tell NXDOMAIN, SERVFAIL, REFUSED, FORMERR, NOTIMP, and NODATA apart (NODATA means RCODE is NOERROR but the target record is empty). If those fields are missing, it draws evidence from dns_raw, dns_raw_inner, message, SOA, EDE, and DNSSEC information, and states the confidence level of the conclusion explicitly.

DNS analysis also compares the DNS Server, city, ISP, monitoring point, CNAME, resolved IP, and target edge zone. Note that if error_code=0 but only an SOA record is returned without A, AAAA, or CNAME records, this only indicates that the current rule evaluation passed; it does not prove that the target record resolved normally.

4. APM Evidence Determines the Backend Boundary

If total synthetic monitoring duration rises while DNS, TCP, and TLS stay normal, and the server span rises with it, the problem lies in the backend service.
If a specific business sub-Span within the server span accounts for most of the duration, the application's internal processing is slow.
If the first concrete error, or most of the duration, appears in a downstream Dependency Span, the problem lies in that downstream dependency.
If the client span is markedly slow while the server span is very short, the problem more likely lies in the path before the application entry, the gateway, the CDN, or an uninstrumented component.
If DNS, TCP, or TLS fails and there is no server span, the request never entered the application.
If a trace_id exists but APM has no data or the Spans are incomplete, the evidence is insufficient, and you cannot conclude from it that the backend is healthy.

The gap between the client span and the server span is not pure network time; it may also include time spent in the CDN, gateway, ingress proxy, queueing, and uninstrumented components. The final conclusion therefore has to draw on synthetic monitoring stage fields, target access evidence, and Span parent-child relationships together.

6. How Cross-Domain APM Analysis Works

Figure 10: Analysis flow from a synthetic monitoring task into the APM call chain

A full cross-domain APM analysis takes four steps:

Starting from the synthetic monitoring task, the Agent uses UModelSearch to locate the synthetics.task entity, confirming the task type, target address, Workspace, and data coordinates.
The Agent queries the synthetic monitoring SLS samples to complete the analysis on the synthetic monitoring side; if the samples carry Trace evidence, it extracts a representative trace_id.
The Agent uses the trace_id to query the APM Trace, retrieving the server span, business processing Span, and downstream Dependency Span for the same request within the backend application.
The Agent aligns the evidence from the synthetic monitoring side and the APM side to form a cross-domain fault demarcation conclusion.

Evidence alignment follows these rules:

Synthetic monitoring is slow and the APM server span slows with it: the problem most likely lies in the backend service.
Synthetic monitoring is slow and the APM downstream Dependency Span slows with it: the problem most likely lies in a backend downstream dependency.
Synthetic monitoring is slow but the APM server span is very short: the problem leans toward the network link, the ingress layer, the gateway, or the path before the application entry.
The DNS, TCP, or TLS stage fails and there is no server span: the request never entered the application, so look first at the path before the application entry.
Trace evidence exists but no backend server span can be found: you cannot conclude that the backend is healthy. The Agent can only flag "insufficient APM evidence" and check Trace reporting, sampling, Region, Workspace, and APM integration.

The APM linkage presents evidence as two link diagrams: one shows the external synthetic monitoring access path, the other the backend Trace path, and trace_id aligns them as the same request. Together they show whether the request reached the service entry point and, once it was inside the application, exactly which Span failed.

Figure 11: Aligning the synthetic monitoring access path with the backend APM Trace via trace_id

7. From "Seeing the Anomaly" to "Knowing What to Do Next"

The value of STAROps is not longer reports; it is a shorter distance from alert to action. It organizes task configuration, raw SLS samples, monitoring point distribution, protocol stages, and APM Traces into one continuous chain of evidence. Users can start from a taskId, task name, or Workspace and work out step by step whether an anomaly comes from a global outage, a regional network, a single ISP, a specific monitoring point, the task configuration, or the backend application and its downstream dependencies.

For experienced engineers, these diagnostic capabilities cut down the constant switching between the Synthetic Monitoring console, SLS, and APM to piece evidence together by hand. For anyone handling a synthetic monitoring fault for the first time, they turn an expert's troubleshooting sequence into a repeatable analysis flow. At Workspace level, the Agent can also pick high-risk targets out of a large pool of tasks and keep running high-frequency monitoring, daily diagnosis, and weekly reviews.

Appendix: Glossary of Terms

Term	Definition
Synthetic Monitoring	Simulates user access from distributed probe nodes to proactively check the availability and performance of target services.
STAROps Agent	An intelligent analysis unit responsible for invoking tools, querying data, aggregating evidence, and generating diagnostic conclusions.
Workspace	A mutually isolated set of observable resources and data scopes.
taskId	The ID that uniquely identifies a synthetic monitoring task.
UModel	A unified model describing observable entities and their relationships, used to locate analysis targets and navigate across domains.
SLS	Alibaba Cloud Simple Log Service (SLS), used to store and query synthetic monitoring execution samples and related logs.
Project / Logstore	A Project is a resource isolation space in SLS, and a Logstore is a data container within it that stores a specific category of logs.
SPL	A query language used to filter, aggregate, and transform log data.
synthetics.task	The entity type in UModel that represents a synthetic monitoring task.
APM	Application Performance Monitoring, used to observe the execution of requests after they enter the backend application.
Trace / Span	A Trace is the complete call chain of a single request, and a Span is an individual processing step within that chain.
same_as	The UModel relationship that associates a synthetic monitoring task entity with its corresponding synthetic monitoring APM service.

Conclusion

Synthetic Monitoring captures the external user experience before a request reaches the service, APM observes how the request executes once inside the application, SLS preserves the factual evidence, and UModel locates the objects and connects the different data domains. STAROps organizes these four into a continuous chain that runs from task discovery through sample aggregation and protocol diagnosis to backend fault demarcation.

This practice turns Synthetic Monitoring from a system that only displays abnormal metrics into an analysis entry point that explains anomalies and supports decisions. Instead of a raw list of fields, users get verifiable answers: where the request stopped, who the anomaly affected, which side owns the fault, what the evidence is, and what to tackle first.

From Reinventing the Wheel to Embedding STAROps via OpenAPI: Why NIIMBOT Chose a Cloud AIOps Foundation

ObservabilityGuy — Fri, 04 Sep 2026 02:00:02 +0000

NIIMBOT integrated Alibaba Cloud's STAROps and UModel via OpenAPI into its SRE platform for automated, cross-domain AIOps diagnostics.

Background

When a technically mature team has moved its entire stack to the cloud and already built a complete observability stack, the key question is no longer "can we build it?" but "what should we build ourselves, and what should we hand to the cloud?" That was the decision facing NIIMBOT. With an established in-house SRE platform, should the company keep building its own observability data foundation and operations digital twin, or adopt a more complete set of ready-made capabilities?

NIIMBOT is dedicated to making things easier to manage through innovation. Since its founding in 2012, the company has continued to advance smart label printing hardware, cloud printing service platforms, and enterprise performance management systems, making label printing smarter and easier. NIIMBOT products are now used across retail, telecommunications, office settings, industry, healthcare and laboratories, and homes, reaching more than 22 million users worldwide.

Business Challenges

As the business expanded rapidly around the world, its systems grew exponentially in scale and complexity, pushing operational complexity to a new level. NIIMBOT runs its business systems on Alibaba Cloud and had already assembled a comprehensive observability stack: Real User Monitoring (RUM) for frontend experience data, Managed Service for Prometheus for container and cloud resource metrics, Application Real-Time Monitoring Service (ARMS) for application performance tracing, and Simple Log Service (SLS) for logs. A self-managed Grafana instance connected to these cloud data sources to provide unified dashboards. Even with this extensive toolset, three structural problems remained.

(1) No Clear Global Topology: Internal Calls and External Dependencies Sat in Separate Graphs

The business systems ran in the cloud, but one question remained hard to answer: which APIs does an application call, and which cloud resources does it depend on? ARMS showed some internal API call relationships, while RDS instances, Redis instances, message queues, and other external dependencies remained scattered across their own monitoring views. The two could not be combined into a single dynamic graph. As the business grew, mapping topology by hand became increasingly inefficient — dependencies documented today could be out of date after next week's release.

(2) Multi-Dimensional Observability Data Was Complete, but Disconnected

Metrics, logs, traces, and events were all being collected. The problem was that collecting everything is not the same as connecting everything. Troubleshooting an incident meant context-switching repeatedly among metric charts, traces, logs, and change events, matching timestamps by hand, with no single thread automatically linking the different data dimensions. The data was there; people still had to piece the clues together.

(3) Alert Noise Made Cross-Domain Root Cause Analysis Take Tens of Minutes

When an incident occurred, every component across the stack — frontend, backend, middleware, database, and containers — raised alerts at once, burying the true source of the problem. Root cause analysis depended heavily on experience: engineers had to decide by instinct where to begin, and locating and analyzing a cross-domain fault routinely took tens of minutes.

These challenges raised a question specific to NIIMBOT. The team was capable of building its own SRE platform, but it also understood the cost of automatically correlating multi-dimensional observability data and maintaining a dynamic, real-time view of topology. Both required substantial investment and continuous upkeep. Was this particular wheel really worth building in-house?

Solution: A Multi-Dimensional Data Foundation, a UModel Digital Twin, and OpenAPI Integration with the In-House SRE Platform

After technical discussions with Alibaba Cloud, NIIMBOT found that the Alibaba Cloud observability system already provided the two capabilities it needed most: automatic correlation across observability data dimensions and dynamic awareness of end-to-end topology. The correlation coverage was broad, and the topology updated automatically — exactly what the team had planned to build. NIIMBOT therefore decided to stop reinventing this wheel. Alibaba Cloud would provide the observability data foundation and operations digital twin; the in-house SRE platform would call STAROps diagnostics through OpenAPI and focus on orchestration and closed-loop workflows tailored to NIIMBOT's business. The solution was implemented in four layers.

(1) A Unified Multi-Dimensional Observability Data Foundation: Bringing Four Existing Collection Capabilities onto One Data Plane

Rather than replace the RUM, Prometheus, ARMS, and SLS deployments already in use, NIIMBOT used Cloud Monitor 2.0 (CMS 2.0) to collect, store, view, and analyze metrics, logs, traces, events, and changes in one place. Frontend experience data from RUM, container and cloud resource metrics from Prometheus, application performance and traces from ARMS, and business logs from SLS now converge on the same data plane. This provides a consistent foundation for topology modeling and intelligent diagnostics. In short, the data no longer sits in separate silos; it enters a shared correlation layer with consistent definitions.

(2) The UModel Operations Digital Twin: Automatic End-to-End Topology Modeling with Real-Time Updates

This capability is the main reason NIIMBOT shifted from building to adopting. UModel models business systems automatically and constructs an end-to-end topology spanning internal API calls and external cloud resource dependencies — frontend → backend → middleware → database → containers — in a single graph. The graph updates in real time as applications are deployed, dependencies change, or workloads scale, with no manual upkeep. Clicking any node brings up its associated observability data. The topology is not a static architecture diagram but a living map backed by current data. UModel's mature correlation coverage and dynamic topology awareness spare the team the ongoing work of building and maintaining these models itself.

(3) STAROps Intelligent Diagnostics: Automatic Cross-Domain Root Cause Analysis on the UModel Topology

With a unified data foundation and the UModel topology in place, STAROps can perform root cause analysis (RCA) automatically when an incident occurs. It follows upstream and downstream relationships in UModel, finds relevant nodes, retrieves their observability data, and quickly returns investigation steps and analysis results. What once required engineers to guess where to begin and query multiple systems by hand becomes AI-driven, cross-domain correlation. In practice, STAROps performs reliably on single-domain analysis of infrastructure resources, such as resource utilization levels for a Pod or an anomalous cloud resource metric. In more complex cases — frequent garbage collection in a Pod, for example, which spans the container layer and application-level JVM metrics — STAROps can start from application monitoring, follow the topology to the JVM metrics, and pinpoint the root cause. This is the capability that turns "the data is there, but people assemble the clues" into "AI connects the data dimensions automatically."

(4) STAROps Integrates with the In-House SRE Platform: Embedding Diagnostics via OpenAPI

NIIMBOT engineers no longer need to leave their familiar SRE platform for another console when troubleshooting. Through OpenAPI, the in-house platform calls STAROps directly as a diagnostic engine and receives a live stream of its analysis and diagnostic results. With one click inside their own SRE platform, engineers can follow the STAROps reasoning process and see its conclusions in real time, completing full-domain diagnosis without leaving the platform.

This model embeds capabilities instead of replacing platforms. NIIMBOT gains the full diagnostic power of the Alibaba Cloud observability system while preserving the autonomy of an in-house SRE platform built around its own business logic. The cloud provides the foundation; the customer's platform remains the interface.

Business Value: Let the Cloud Handle the Heavy Lifting and Refocus on the Business

(1) One-Click, Closed-Loop Full-Domain Diagnosis Inside the In-House SRE Platform

Previously, troubleshooting a cross-domain fault meant switching repeatedly among ARMS, Prometheus, SLS, and Grafana. Engineers matched timestamps and assembled clues by hand, and root cause analysis routinely took tens of minutes. Now, a single click in the NIIMBOT in-house SRE platform triggers diagnosis of a business system issue. STAROps follows the UModel topology, correlates multi-dimensional observability data across domains, and returns the root cause analysis. The operations team no longer has to sift through each data dimension or hunt for clues layer by layer.

(2) No More Reinventing the Wheel: A Mature Team Puts Its Effort Back into the Business

NIIMBOT once had to commit its own engineering resources to building and maintaining general-purpose operations capabilities such as topology modeling and multi-dimensional data correlation. That costly, maintenance-intensive work is now handled by the Alibaba Cloud observability system and UModel. Freed from reinventing the wheel, the team can focus again on cloud resource management and system architecture optimization — work that contributes more directly to its business. For mature teams, deciding what to build and what to hand to the cloud is a real strategic choice; NIIMBOT has made its own.

(3) Operations Evolves from Reactive Firefighting to Proactive Management

The operations team once spent its days waiting for alerts, chasing faults, and conducting post-incident reviews. With UModel's dynamically updated end-to-end topology and AI-driven analysis of multi-dimensional observability data, the team has moved from reactive response to proactive inspection and earlier problem detection. While keeping business systems stable, its role has evolved from incident responder to steward of system health, completing its transition to an AIOps operating model.

Looking Ahead: From Diagnostics That Work to Diagnostics That Know NIIMBOT Better

As STAROps becomes embedded in the in-house SRE platform, more core business systems will gain end-to-end topology modeling and intelligent diagnostics, extending the living map and one-click cross-domain diagnosis across the entire business. Each new system onboarded strengthens the AIOps capability of the NIIMBOT in-house SRE platform: wherever the business expands, AIOps coverage follows.

(1) From Trace-Guided Correlation to Automatic Analysis from Any Entry Point

STAROps already pinpoints root causes by correlating across domains along the UModel topology. The next step is to make the result independent of where an investigation begins. Whether an engineer starts from application monitoring or from an infrastructure resource such as a Pod or container, the system should extend upstream and downstream automatically and connect every observability data dimension. The goal is more seamless, intelligent cross-domain correlation, with root cause analysis that no longer depends on choosing the right entry point.

(2) Making the Embedded OpenAPI Integration Smoother

Streaming the diagnostic process back to the in-house SRE platform through OpenAPI is central to this integration. Guided by NIIMBOT's feedback from day-to-day use, the two sides will continue improving the responsiveness and completeness of the stream, making the analysis shown inside the platform more detailed and coherent — and turning embedded capabilities into a genuinely seamless experience.

Flutter RUM in Practice: Reconstructing the Full Story Behind an AI App's Wait

ObservabilityGuy — Tue, 01 Sep 2026 07:31:52 +0000

In the AI Era, When an App Keeps Spinning, Don’t Blame the API Just Yet

When users report that "the page keeps spinning," "nothing happens after I tap the button," or "the content froze halfway through loading" in a Flutter app, most engineers immediately check API latency, error logs, or crash stacks. But production problems rarely have a single cause. One wait can span a user operation, page state, a network request, on-device rendering, error handling, and calls to native platform capabilities.

Looking at only one type of log can easily lead to an incomplete conclusion:

Check only API logs and you may miss on-device rendering and blocked state updates.
Check only Dart exceptions and you may not see the tap or the network request that preceded them.
Check only page instrumentation and you may not be able to tell which step the user is actually stuck on.
Check only crashes or errors and you may not be able to reconstruct the full path leading up to the problem.

The Alibaba Cloud Real User Monitoring (RUM) Flutter SDK addresses these problems for Flutter apps. Through alibabacloud_rum_flutter_plugin, it collects context in the Dart layer — pages, network, action, LongTask, exceptions, resource snapshots, and custom business fields — and passes that data to the native RUM SDK for reporting and correlation. This article uses these implementation details to show how to turn a single user wait into a traceable, verifiable account of what happened in production.

To Reconstruct an AI App's Wait, Follow the Path Through Flutter's Layers

A typical Flutter interaction path breaks down roughly into the following stages:

The user taps a button 
-> Flutter Action detection 
-> business state machine update 
-> network request or local task execution 
-> response returns 
-> page content refresh 
-> list, rich text, or image rendering 
-> the page reaches a stable state

All the user sees is "it took a long time." What engineers actually need to answer is where the time went: before the tap, before the request, during the request, after the response, or in the on-device rendering stage.

Traditional troubleshooting struggles to answer this question because it has several blind spots.

Visibility gap	Production symptom	Why traditional methods fall short
Behavior gap	No response after tapping a button, repeated taps, accidental triggers	Manual instrumentation can easily leave gaps, and definitions are inconsistent
Network gap	Slow APIs, failed requests, retries on weak networks	With only URL and duration, it is hard to get back to the specific page and user session
Rendering gap	Stuttering on list refresh, slow rich text rendering, brief page freezes	Server-side logs show nothing about main Isolate pressure on the client
Exception gap	State machine exceptions, parse failures, null object errors	The Dart stack lacks the preceding action and resource context
Session gap	Many logs exist, but they cannot be connected into a complete picture of a single user session	Engineers must manually correlate logs by timestamp, device, app version, and page path

The point of Flutter RUM, then, is not to collect a few more SDK event types. It is to build a connected timeline around a real user's experience:

Only when these events land in the same RUM Session can engineers turn "the page keeps spinning" into a set of testable hypotheses.

Connect the Clues Across Layers: Bridge Collection from Dart to Native

A Flutter app's observability path is inherently cross-layer. The Dart layer knows about Widget, Route, Zone, Dio, and business state; the Android, iOS, and HarmonyOS native SDKs are better suited to platform-side reporting, network tracing configuration, and data delivery.

Structurally, the Flutter RUM SDK splits into three layers:

The Dart collection layer preserves Flutter semantics — Dart Zone exceptions, Route lifecycles, Widget taps, Dio requests, and main Isolate blocking. The native SDK handles the standardized events and platform-side reporting.

This division of labor pays off most in complex Flutter scenarios: the Flutter side keeps the semantics of what the user did, what state the page was in, and how content refreshed, while the native and RUM sides place that data into a single session view.

The First Clue Starts with the Tap: Did the Action Reach the Current Session?

"Nothing happens after I tap" is a common report from production users. It can have several causes:

The button is disabled, or the business state machine never advances to the next step.
The tap triggers a request, but the request fails at the network layer or is retried.
The request goes out, then a synchronous task blocks the main Isolate, so the page does not refresh in time.
An automated flow or a system task triggers a duplicate operation, corrupting business state.

The first step here is not to open the API logs. Instead, confirm that the user behavior actually reached the same session.

User behavior in Flutter is not a native button tap, and it is not a Web DOM click. An operation starts as pointer events and coordinates, so the SDK has to go back into Flutter's HitTest, Widget, Element, and RenderObject system to recover the semantics.

Automatic action detection is not enabled by calling start() alone. The app must wrap the target widget tree in AlibabaCloudActionCapture:

AlibabaCloudActionCapture(
  child: MaterialApp(
    navigatorObservers: [
      AlibabaCloudRUMNavigationObserver(enablePagePerf: true),
    ],
    home: HomePage(),
  ),
);

For critical business operations, do not rely on default control detection alone. Add business semantics through ActionAnnotation:

ActionAnnotation(
  description: 'Submit Button',
  attributes: {
    'screen': 'order_detail',
    'action': 'submit_order',
    'actor_type': 'human',
  },
  child: ElevatedButton(
    onPressed: _submitOrder,
    child: Text('Submit'),
  ),
);

An operation driven by an automated business flow does not necessarily produce a standard pointer event that Flutter can observe. When the app calls a method directly, triggers a background task, or runs a rules engine, the SDK cannot recover the full semantics from a tap alone; if the tap comes from system accessibility or a simulated coordinate tap, it may still look like an ordinary tap. We therefore recommend that the app explicitly report an Action or a custom event at critical points in the flow, and add context such as actor_type.

actor_type	Meaning	Typical scenario
human	Action performed by a human user	Tapping submit, refresh, back, or retry
automation	Automated business flow	Scheduled retries, batch processing jobs, rule-triggered jobs
system	Triggered by an automated system process in the app	Automatic recovery, background refresh, timeout retry

With that in place, RUM has more than a single tap to work with when you investigate "nothing happens after I tap." By correlating the action with business events and subsequent resource, LongTask, and error events, engineers can reconstruct who or what initiated the operation and how it executed.

Where Did the Request Go After the Tap? Resource Reconnects Network Activity to the User Session

When a user says "the page keeps loading," the API is not necessarily slow. Break it into at least these stages:

From the tap to the request going out.
From the request going out to the response coming back.
From the response coming back to the page finishing its update.
Whether the page becomes interactive after it updates.

The Flutter RUM SDK offers two entry points at the network layer: when you use dart:io directly, wrap the global HttpClient through HttpOverrides; when you use Dio, integrate through AlibabaCloudRUMDioInterceptor.

final dio = Dio();
dio.interceptors.add(
  AlibabaCloudRUMDioInterceptor(
    onProvideSnapshots: (requestOptions, response, error) {
      return ResourceSnapshots(
        requestHeaders: {
          'content-type': requestOptions.headers['content-type'] ?? '',
        },
        responsePayload: response?.data is Map
            ? {
                'code': response?.data['code'],
                'requestId': response?.data['requestId'],
              }.toString()
            : null,
      );
    },
  ),
);

For a Flutter app, network monitoring is valuable not because it records one request's latency, but because it puts the API request, the user operation, and the page state back on the same experience path. When a user says "the page keeps spinning," engineers need to know whether the request was even sent, whether the API timed out, whether the server-side path failed, whether the page updated promptly once the response came back, and on which page, which version, and in which session all of this happened.

Where business policies permit, RUM can correlate client-side requests with server-side traces, turning troubleshooting from reading one isolated API log into reconstructing the full path of a user's wait. Requests involving third-party domains, sensitive APIs, or user input must still comply with business security policies and data governance requirements.

Resource latency alone is not enough. For critical business paths, add business fields that help locate the problem:

Field	Suggested source	Purpose
flow_id	Session-level custom property	Links the requests and page events within a single business process
request_id	Resource extension property or server-side trace	Aligns client-side requests with server-side logs
business_stage	Custom event or log	Identifies the business stage in which the wait occurred
result_status	Custom event	Distinguishes success, user cancellation, network failure, and server-side exception
error_stage	Custom event or log	Distinguishes failure stages such as parameter validation, networking, server-side processing, and rendering

These are not built-in fields the SDK promises to collect by default; they are context we recommend you add for business troubleshooting. The SDK provides a stable foundation for collection, reporting, and session correlation. Business semantics still have to be designed by engineers around their own paths and compliance requirements.

The Response Is Back. Why Is the Page Still Stuttering? LongTask Reveals Client-Side Pressure

When a Flutter page refreshes its content, it may trigger state updates, rich text rendering, image loading, long-list diffing, or JSON parsing. If that logic consumes too many on-device resources, the user sees a page that stutters, while the server-side logs look entirely normal.

The Flutter RUM SDK watches for periods when the Dart main Isolate remains unresponsive. When text rendering, list updates, or complex layouts consume too many client-side resources, the SDK can record a LongTask event and place the blocking duration, affected page, and user session on the same timeline.

Engineers do not need to focus on the underlying detection algorithm here. What matters is preserving evidence when the server has returned the content but client-side rendering cannot keep up. Then, when a user reports that "the page froze," the investigation can move beyond the server and network APIs to Flutter's rendering and state-update path.

These events fill in what happens on the client after the server responds. Within a single session, for example, you might see:

Action: Submit
-> Resource: /api/order/submit 200
-> Custom: business_stage = render_result
-> LongTask: 236ms
-> LongTask: 410ms
-> View: OrderResultPage

At that point engineers can form a working hypothesis: the API response is not slow, but the main Isolate is under significant pressure while the page refreshes. From there, keep verifying against data volume, list length, rich text node count, device model, and page structure.

LongTask is therefore best treated as one signal in user-experience troubleshooting. It can point to pressure in client-side rendering or state updates; correlated with action, resource, error, and business events on the session timeline, it helps engineers determine whether the bottleneck is on the client. If LongTask events cluster during a page refresh after the API has already returned, inspect list refreshes, layout calculations, and state-update frequency first.

When a Stutter Is Followed by an Error, Read It in Context

The stack trace may make many Flutter exceptions easy enough to understand. What it does not explain is why the exception occurred for this user, on this page, after this operation.

A state exception, for example, may come from:

The user taps submit 
-> the request goes out 
-> the server returns an unexpected structure 
-> Dart parsing logic enters the exception branch 
-> the Flutter state update fails 
-> an Error is reported

From the exception stack alone, it is hard to tell whether the parameter was invalid, the business API failed, a native capability failed, or the Flutter state machine failed to handle the response.

Flutter exception collection cannot rely on a single entry point. During initialization, the SDK hooks into several exception paths:

It catches unhandled exceptions inside the Zone through runZonedGuarded.
It takes over FlutterError.onError to handle synchronous Flutter framework exceptions.
It takes over PlatformDispatcher.instance.onError to handle uncaught exceptions at the platform dispatch layer.
It preserves the existing handler chain, minimizing interference with the application's current error-handling logic.
It provides onRUMErrorCallback so you can decide whether to keep reporting.
It uses setDumpError to control whether Flutter errors still go to the console.

For standard cases, just use start():

void main() {
  AlibabaCloudRUM().start(MyApp());
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await AlibabaCloudRUM().initialize();
  runApp(MyApp());
}

Note that if you choose initialize() and call runApp() yourself, some capabilities that depend on post-startup application state must be enabled separately at an appropriate time. For example, if you need LongTask detection, enable it after runApp() using the current public API:

await AlibabaCloudRUM().initLongTaskDetection();

For business troubleshooting, the exception alone is not enough. We recommend adding the following semantics:

Field	Meaning	Example
`business_stage`	Current business stage	`submit`, `pay`, `render_result`
`request_id`	Request ID	Correlates the request with server-side logs or traces
`error_stage`	Failure stage	`param`, `network`, `server`, `render`
`result_status`	End reason	`success`, `user_cancel`, `server_error`

When an error occurs, it is no longer just a Dart stack. Engineers can analyze it alongside the preceding action, resource, page state, and business stage.

Still Not Enough Clues: Add Context with Resource Snapshots While Respecting Data Boundaries

When investigating API problems, engineers usually want more context, such as error codes, request IDs, server response status, and whether required parameters are missing. But headers and payloads may contain user input, authentication data, or sensitive business fields, so they cannot be collected in full by default.

The SDK therefore uses an explicit Provider mechanism:

No header or payload is collected by default.
Collection happens only after you set a ResourceSnapshotProvider or Dio's onProvideSnapshots.
You are responsible for filtering, data masking, and ensuring compliance.
The SDK enforces size limits after the Dart-layer Provider or Dio callback returns: headers are measured by their JSON UTF-8 size and dropped if they exceed 64 KB; payloads are measured in UTF-8 bytes and truncated if they exceed 150 KB.

For critical requests, keep only the fields that help diagnosis and contain no sensitive content, for example:

ResourceSnapshots(
  requestHeaders: {
    'content-type': requestOptions.headers['content-type'] ?? '',
  },
  responsePayload: response?.data is Map
      ? {
          'code': response?.data['code'],
          'requestId': response?.data['requestId'],
        }.toString()
      : null,
);

Resource snapshots are meant to add troubleshooting context, but the size limits are not a substitute for your own data masking and compliance review. When integrating, prioritize the minimum diagnostic fields such as error codes and request identifiers, and avoid uploading user input or full business responses.

This is a deliberate design trade-off: the SDK provides a channel for collection and reporting, but it does not read application payloads on its own. This prevents monitoring logic from affecting the application's data flow or introducing compliance risks.

When Is an AI Chat Page Really Usable? Use View Metrics to Separate a Slow Page from a Slow Task

A Flutter page is usually not a static page that loads once. After a user opens an order, payment, content detail, or workbench page, the app may need to show the page container first, then load API data, render lists or rich text, and finally make key operations such as submit, refresh, and filter available. Page metrics are therefore better suited to answering a few business questions: did the page appear promptly, is the key content visible, are the main operations available, and can the user finish the current task quickly.

The Flutter RUM SDK collects standard routing scenarios through AlibabaCloudRUMNavigationObserver:

MaterialApp(
  navigatorObservers: [
    AlibabaCloudRUMNavigationObserver(
      ignoreRoutes: ['/splash'],
      enablePagePerf: true,
    ),
  ],
  home: HomePage(),
);

For non-standard page structures such as IndexedStack, PageView, and Tab containers, you can also use the manual API:

AlibabaCloudRUM().startView('OrderDetailPage');
AlibabaCloudRUM().stopView('OrderDetailPage');

Page performance collection is organized around a single Route lifecycle and focuses on the following metrics:

Metric	Meaning	How it is collected on the Flutter side	Questions it answers
TD	Transition Duration	Derived from TransitionRoute.animation state; in the current public implementation it is not necessarily reported as a separately queryable field	Whether entering a key page is slowed down by the page transition
FP	First Paint (FP)	Captured through WidgetsBinding.instance.addPostFrameCallback	Whether the page container appears quickly
FCP	First Contentful Paint (FCP)	Traverses RenderObject to detect content such as RenderImage, RenderParagraph, and TextureBox	When the first piece of business content, such as order, list, or detail content, appears
TTI	Time to Interactive (TTI)	Based on the effective element coverage rate; a fallback timing policy applies when the threshold is not reached	When the main operations, such as submit, refresh, and filter, become available

FP, FCP, and TTI in Flutter are estimates based on Flutter rendering and page structure, so they should not be interpreted in the same way as browser page metrics. For PlatformView, custom-painted widgets, or complex page containers, validate each metric against the page structure. Page performance data is reported through extension fields when the View event ends; the fields ultimately available for queries depend on the View extension map and each platform SDK's support.

For complex pages, read page metrics together with business stages. For example:

View: OrderDetailPage
-> FP / FCP / TTI
-> Action: Submit
-> Resource: /api/order/submit
-> Custom: business_stage = render_result
-> LongTask: page render

That lets you separate "the page itself opens slowly" from "the business processing is slow after the page opens."

After the Evidence Reaches RUM: Let STAROps Answer "Why Is the AI Still Waiting?"

Once RUM data reaches the platform, troubleshooting should not stop at handwritten queries. Start with the troubleshooting question, then let the observability platform help structure the analysis path.

Within the capabilities currently available in Cloud Monitor 2.0 (CMS 2.0), STAROps can serve as an assisted analysis entry point. CMS 2.0 refers here to the new generation of Cloud Monitor console capabilities, while STAROps provides assisted analysis of observability data. Product names, scope, and entry points are subject to current console availability.

For a Flutter app, the question is no longer "which table should I query?" It sounds much more like a real production troubleshooting question:

Which pages had abnormal wait times in the past hour? 
In sessions where nothing happened after the submit button was tapped, did LongTask events or slow requests appear at the same time? 
Are API errors concentrated in a specific version or a specific class of device? 
After a version upgrade, did Resource errors on the page and TTI increase at the same time?

STAROps helps by organizing these natural-language questions into an analysis path:

Start from the symptom. First describe the problem: "the page keeps spinning," "nothing happens on tap," "the API fails."

Narrow the impact. Use application version, operating system, device, page, region, and time window to define the affected scope.

Correlate session events. Link the action, page performance, resource, error, LongTask, and business events from the same user session.

Develop working hypotheses. Possible causes include client-side blocking, a slow API, retries along the path, rendering failures, or resource errors concentrated in one version.

Inspect individual samples. Return to a specific session, page path, request path, and exception context to test each hypothesis.

What STAROps can do, which entry points are available, and how well the analysis works all depend on what the console currently supports, and the results also depend on the completeness of the underlying RUM data. If pages are named inconsistently, Actions lack business semantics, resource snapshots are not masked and enriched as needed, or tracing is not connected end to end, assisted analysis can only see part of the picture. Automatic collection on the SDK side and semantic enrichment on the business side remain the foundation.

Bring the Troubleshooting Path into the SDK: Start with One Key Page

Trying to cover every Flutter page, network request, and business flow from day one increases integration costs and makes definitions harder to standardize. A better approach is to pick one representative critical page and establish a minimal end-to-end loop first.

1. Get RUM Running First

Standard case:

void main() {
  AlibabaCloudRUM().start(MyApp());
}

Custom startup flow:

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await AlibabaCloudRUM().initialize();
  runApp(MyApp());
}

2. Then Wire Up Pages, Network, and Behavior

Mark the key pages with AlibabaCloudRUMNavigationObserver or startView/stopView.

Collect the core business requests with AlibabaCloudRUMDioInterceptor or HttpOverrides.

Use AlibabaCloudActionCapture and ActionAnnotation to detect key operations such as submit, refresh, back, and retry.

3. Once the Basic Data Is Flowing, Add Business Semantics

Once view, action, resource, LongTask, and error collection are in place, RUM already records what the user did, whether requests succeeded, whether the page stuttered, and whether an exception occurred.

But these technical events do not fully explain the business process. Developers still need to know which flow the events belong to, which stage the user is waiting on, and how the operation ended. The application only needs to add enough context to answer those questions:

Context to add	Reference fields	Questions it answers
Flow correlation	flow_id, request_id	Whether these pages, operations, requests, and exceptions belong to the same flow
Execution stage	business_stage	Whether the wait occurs during the request, processing, parsing, or rendering stage
Execution result	result_status, error_stage	How the flow ended, and at which stage the exception occurred

In an AI conversation scenario, flow_id can map to one conversation or task, and business_stage can separate stages such as calling the model, streaming the response, and rendering the page. Design the field names and values around your actual business workflow; they are not built-in fields provided by the Flutter RUM SDK.

Design fields so that they explain problems; you do not need to cover every state change. Once these semantics are in place, the next step is to check that they show up in the same session as view, action, resource, LongTask, and error events.

4. Finally, Check That One Session Is Complete

After integration, do not just check whether individual metrics appear. Verify that RUM captures a complete user timeline:

View: OrderDetailPage
-> Action: Submit
-> Resource: /api/order/submit
-> Custom: business_stage / result_status
-> LongTask: page render
-> Error: optional

Once RUM can stitch this trace together, extend it step by step to more pages, more business processes, and more business semantics.

Back to the Beginning: Reconstruct the Wait, Don't Just Add More Logs

Production experience problems in a Flutter app usually cannot be explained by one API, one stack trace, or one tap. A report that "the page keeps spinning" may involve client-side interaction, a network request, server-side processing, page rendering, main Isolate blocking, and exception handling. Server-side logs alone cannot show the client-side page state or rendering behavior; a Flutter exception stack alone cannot show the preceding action and request path.

The Flutter RUM SDK brings these scattered events back into one user session: what the user did, whether the request went out, whether the API failed, whether the page stuttered, and whether an exception occurred on the same path. Instead of a pile of isolated logs, engineers get a coherent record they can trace, correlate, and review.

This approach is already implemented in the Alibaba Cloud RUM Flutter SDK. For the exact APIs, fields, and supported platforms, refer to the official release and the integration documentation. There is still room to go further in areas such as complex gesture detection, LongTask analysis, standardized business fields, compliance policies for resource snapshots, and field consistency across platforms.

For mobile developers, the goal of observability is not to collect more data. It is to break a real wait into a verifiable trace so engineers can understand the problem faster.

Learn more: https://www.alibabacloud.com/help/cms/cloudmonitor-2-0/access-to-mobile-applications-developed-through-flutter

From a Single Alert Card to One-Click RCA: How Tastien Built a Closed-Loop AIOps Practice Across 10,000+ Outlets

ObservabilityGuy — Fri, 28 Aug 2026 07:35:07 +0000

This article introduces how Tastien built a closed-loop AIOps system to unify alerts, enable AI-driven root cause analysis, and cut MTTR by 65% across its 10,000+ outlets.

Founded in 2012, Fuzhou Tastien Catering Management Co., Ltd. is a restaurant chain built around the "hand-rolled, freshly baked" Chinese burger. Drawing on traditional Chinese pastry craft, Tastien developed a distinctively textured hand-rolled, freshly baked burger bun and created an entirely new category: the Chinese burger. Strong R&D, consistently high product quality, and a convenient, attentive service experience have made Tastien one of China's leading restaurant brands. Today it operates more than 10,000 outlets across over 300 cities, with a large and loyal consumer following.

At a chain of this size, operational systems span dozens of service chains — outlet POS, supply chain, Customer Relations Management, online ordering, and more. In the AI era, as the business grows and iterates at speed, monitoring covers ever more ground and teams configure ever more monitoring and alerting — which brings problems of its own. Alerts arrive from many sources: CloudMonitor, ARMS, Simple Log Service, custom business alerts, and certificate alerts. Because that alert data sits in separate systems with different data formats, notification methods, and handling flows, the cost of configuring alerts, delivering notifications, responding and investigating, capturing knowledge, and closing the loop keeps climbing.

The Business Challenge: What Happens After the Alert Determines O&M Efficiency

In a multi-business cloud environment, companies rarely lack monitoring tools. What they lack is the ability to turn an alert into a management event that teams can work on together, track, and reuse. One fault can fire alert after alert until the important details are buried in the flood. On-call engineers switch between platforms and stitch the context together from experience. AI returns a diagnosis, but with no human feedback, no one can tell which analyses were right and which need work.

Traditional monitoring solved the problem of detecting faults. Alert management, and the collaboration that follows an alert, still runs into the same recurring problems:

Alerts scattered across sources: Alerts come from many places, and the data lives in four or five different systems. Each monitoring system needs its own contacts, notification methods, and delivery policies — expensive to maintain and easy to get wrong. Worse, alert content and status are expressed differently in each system, so on-call engineers struggle to read the alert status quickly, locate the problem, or notify the right owner in time.
Heavy duplicate noise: A single alert rule usually covers many instances, sometimes dozens. When one fault trips them all at once, a flood of duplicate Lark messages and phone calls follows within minutes — an alert storm. Duplicates add to the judgment and handling load on on-call engineers and drown out what actually matters, so critical alerts get missed.
Diagnosis that depends on individual experience: When an alert fires, the root cause may lie in a metric, a Simple Log Service log, an application trace, or an operational system. Investigating means switching back and forth across platforms and manually correlating monitoring, application, container, and log context — slow work that rests heavily on how much the on-call engineer happens to know.
No closed loop, no accumulated knowledge: Alert recovery is not the same as closing the loop. Without structured capture of root causes and solutions, past experience is hard to retrieve and reuse, and it never feeds back into the next diagnosis.

In our view, what happens after the alert is what decides O&M efficiency. Solving these problems calls for a single intelligent AIOps platform that ingests alerts from every source, standardizes them, and orchestrates them — delivering each alert to the right O&M engineer, helping them locate and fix problems fast, limiting the impact, and capturing what was learned so the alert loop keeps improving. The goal was never to add one more alert page, but to give every alert a complete lifecycle.

The Solution: One Alert Entry Point, an AI Feedback Loop, and a Digital Worker Matrix

To tackle scattered sources, duplicate noise, slow root cause work, and lost knowledge, Tastien built on Alibaba Cloud CloudMonitor 2.0 (CMS 2.0) and the full-stack AIOps platform STAROps. The Unified Alert Center in CMS 2.0 became the single entry point, a low-code workflow was added to the existing O&M platform as the alert orchestration layer, and together they close the loop:

CMS 2.0 multi-source alert ingestion → low-code workflow orchestration → automatic convergence → Lark collaboration → STAROps AI-assisted Analysis → root cause information added → AI rated → knowledge captured

(1) Unified Alert Management in CloudMonitor 2.0 + Workflow Orchestration: One Entry Point for Every Alert Source via Event Integration and Subscription

Tastien routes ARMS, CMS 1.0, Simple Log Service alerts, Prometheus, and custom alerts into the Event Center through the event integration capability of CMS 2.0. A workflow then orchestrates the rules that normalize them into management events:

Webhook ingestion → source detection → field standardization → convergence decision → group routing and on-call scheduling → card generation or update → write to data store → tiered escalation and feedback.

The workflow picks a parse branch by alert source and transforms the title, level, resource, status, and other information into one common context. It then invokes alert convergence and group routing to decide whether to send a new card, update an existing one, or close it out on recovery, and it applies the matching notification and escalation policy for P1, P2, or P3. Claims, status updates, and root cause feedback raised on the Lark card flow onward through the same orchestration chain. This keeps the visual orchestration flexible while keeping long-lived state, consistency, and historical facts out of the workflow. Adding a new alert source usually means adding a parse branch and reusing the shared downstream stages, with no need to rebuild the whole handling chain.

(2) Lark Alert Cards + AI Root Cause Analysis: Claim, Ask, and Track End-to-End on One Card

The platform keeps real-time status, the collaboration entry point, and historical facts apart: real-time status drives alert convergence and card updates, Lark carries the live collaboration, and historical data preserves confirmed root causes, feedback, and handling records.
On-call engineers claim an alert, view its status, and reference the card to ask the AI what caused it — all from the card itself. The system pulls in the current alert context automatically, so no one has to restate the situation or search across platforms.
Typical scenarios delivered on the Lark alert card:
Ask about O&M alerts: describe the query in natural language, such as today's alert volume, unclaimed alerts, or top alert rules
Create Alert Rules: create cloud alerts and custom alert rules in natural language
Ask follow-up questions from an alert card: reference the card, skip the recap, and let the AI pull in the context
One-click root cause analysis: reference the card to ask the AI, which queries historical root causes and invokes STAROps root cause analysis to reach a conclusion, streaming the diagnosis in real time so the process stays transparent
Human-AI collaboration: interactive execution, with manual review for high-privilege commands

This turns the alert card from a notification into a handling entry point: who owns the problem, how far the analysis has progressed, and what the root cause turned out to be all accumulate around the same management event.

(3) The AI Feedback Loop: Every Diagnosis Captured as a Triplet of Context, Human-Confirmed Root Cause, and Rating

Whether AI root cause analysis is reliable cannot be settled by impressions on the product side; it takes data. Tastien embedded a minimal but complete feedback mechanism into the handling chain:

Step 1 · AI produces a diagnosis: after an on-call engineer references the alert card and asks a question, the AI produces a diagnosis based on the injected management event context and the historical root cause repository for that service.

Step 2 · The engineer fills in the actual root cause: once the management event has recovered (or once the root cause is confirmed during handling), the on-call engineer describes the actual root cause through the "Fill in Root Cause" entry on the card — free text, but it must state clearly what actually happened.

Step 3 · Rate AI accuracy: after the root cause is filled in, the system opens a rating entry with only two options — "AI is accurate" or "AI is inaccurate". If the engineer selects "AI is inaccurate", they can add an optional one-line note (for example, "AI attributed the root cause downstream; it was actually an upstream traffic problem").

As these records build up, they deliver two direct benefits:

A historical root cause repository: the next time a similar alert hits the same service, the AI reads that service's historical root cause records first in its Step 1 diagnosis, drawing on real past root causes rather than reasoning from current metrics alone. The more it is used, the closer its conclusions track the customer's actual business.
Diagnosis accuracy statistics and scenario-based improvement: the team can measure AI accuracy by service and alert_type, then target improvements at specific scenarios.

(4) The STAROps Digital Worker Matrix: Skill Encapsulation, Mission Orchestration, and One-Click RCA

On top of the alert loop, Tastien built a second layer of capability with STAROps: encoding the troubleshooting steps held in senior SREs' heads as Agent capabilities the platform can schedule.

Skill Encapsulation — Turning Troubleshooting SOPs into Reusable Capability Units

Each Skill defines the complete troubleshooting path for one specific fault scenario: trigger conditions (which alert or metric combination should invoke the Skill), data collection steps (which metrics, logs, and traces to query, and in what order), judgment logic (which metric combination points to which root cause), and output format (a structured diagnosis plus recommended actions).

Mission Orchestration — Running Skills in Parallel to Form an Inspection Matrix

A single Skill handles diagnosis for a single scenario, but daily inspection needs dozens of Skills running in parallel under line-of-business and time-window policies. Using the Mission (long-running task) capability in STAROps, Tastien orchestrates multiple inspection Skills into continuously running digital workers. After each inspection round, the Agent outputs a structured inspection report: normal items collapsed, abnormal items highlighted with a diagnosis and recommended actions. The SRE's day shifts from "watching monitoring dashboards in shifts" to "reading the inspection report each morning and handling the items flagged in red".

Taken together, what Tastien built is not one more alert page. It turns alerts from messages into management events, AI output into data that can be rated, and a one-off incident response into a reusable O&M asset.

Only when every alert is grouped correctly, handled promptly, reviewed in full, and fed back into the next diagnosis does an O&M platform move from a notification tool to an engineering system that keeps learning.

Results: From an Engineering Loop to Measurable Gains

Significant reduction in alert noise: Through unified alert convergence and AI-based noise reduction in CloudMonitor 2.0, the alert compression rate reached 95%, and the invalid alert rate decreased by 15%.
Substantial improvement in incident resolution efficiency: With AI-assisted root cause analysis and end-to-end collaboration through Lark, the Mean Time to Recovery (MTTR) was reduced by 65%.
Effective accumulation of O&M expertise: A historical root cause library covering more than 40 business-specific scenarios has been established, enabling sustainable reuse of operational experience and continuous evolution of AI models.

From 'Seeing' to 'Self-Healing': Chanjet's Observability and AIOps Practice

ObservabilityGuy — Wed, 26 Aug 2026 02:56:17 +0000

This article introduces how Chanjet transformed its traditional O&M system into an AI-driven AIOps and observability platform to achieve proactive fault prevention.

1. Background and Challenges

As a leading provider of financial, taxation, and business cloud services for micro and small enterprises in China, Chanjet operates across five major product lines and nine clusters, serving millions of businesses. Currently, its core business has fully completed its SaaS transformation and cloud-native transformation, deploying a multi-tenant, multi-center architecture. With continuous customer growth and growing business complexity, the shortcomings of the original traditional O&M monitoring system became increasingly apparent. These primarily manifested in three major challenges: lack of visibility, unmanageability, and an inability to resolve issues, making a comprehensive system upgrade urgent.

(1) Lack of Visibility: Falling into the "Metric Trap" with Delayed Perception of User Experience

The infrastructure monitoring system for underlying resources like CPU, memory, and disk was established early and highly mature. However, with the rapid development of the SaaS and multi-tenant models, observability tailored to user experience was sorely lacking. Issues directly impacting customer experience—such as domain access anomalies, slow API responses, functional errors, and page freezes—could not be proactively identified by traditional monitoring. Often, troubleshooting only began after customer complaints or public feedback. The team defined this issue as the "Metric Trap": all underlying monitoring metrics appeared normal, yet the end-user experience had significantly degraded.

(2) Unmanageability: Inefficient Alert Floods and Prolonged Troubleshooting

As the system scaled, alert volumes exploded exponentially. A single underlying storage fluctuation could trigger hundreds of correlated alerts, making it difficult for O&M personnel to quickly pinpoint the root cause of the core failure. Meanwhile, insufficient alert tiering and aggregation capabilities meant that critical emergencies were easily drowned out by a flood of low-priority alerts. Previously, it took an average of over 10 minutes from receiving an alert to confirming the root cause. This prolonged the end-to-end failure recovery cycle, leaving significant room for optimization across mean time to identify (MTTI), mean time to know (MTTK), mean time to fix (MTTF), and mean time to verify (MTTV).

(3) Inability to Resolve: Reliance on Experience and Insufficient Proactive Defense

Once a fault was located, emergency mitigation relied heavily on the personal experience of senior O&M staff. Although the team had outlined standardized mitigation strategies such as rate limiting, service degradation, and failover, these were never implemented as one-click automated runbooks within the complex multi-center architecture and multi-tenant environment. Furthermore, proactive risk prevention capabilities were weak. Many failures could have been avoided through proactive inspections and configuration validation, highlighting the urgent need for a systematic prevention mechanism.

In light of these challenges, Chanjet set clear upgrade goals: to increase the overall service level agreement (SLA) from 99.9% to 99.995%, and to build O&M capabilities that enable 99% proactive fault prevention and 10-minute emergency mitigation. Centered on user experience, the ultimate aim was to steadily improve customer satisfaction through a comprehensive reconstruction of the technical architecture and O&M model.

2. Building the Observability System — "Seeing" Issues Clearly and Comprehensively

To address the "lack of visibility" pain point, Chanjet combined business characteristics with application tiering standards (Tier-1, Tier-2, and Tier-3 applications) to build a five-layer integrated monitoring model, establishing layered, comprehensive, and precise observability capabilities.

Infrastructure monitoring: Covers foundational metrics such as CPU, memory, disk, ports, and network, securing the baseline of system operations.
Middleware monitoring: Focuses on middleware like Redis, databases, and message queues, monitoring core data such as resource utilization, connection counts, and bandwidth to detect component bottlenecks and connection anomalies at the earliest opportunity.
Application performance monitoring (APM): Collects operational metrics including GC frequency, thread status, Pod response times, and blocked threads to grasp the real-time health status of applications.
Business monitoring: Captures business-level error signals from logs, such as database connection exceptions, memory overflows, and rate limiting events, directly addressing fundamental issues in business operations.
User experience monitoring: Based on access logs for domains and core APIs, it monitors anomalous status codes like 500, 499, and 302, as well as sudden spikes in response latency, evaluating service quality from the user's perspective.

The data collection system of this five-layer model highly aligns with the design philosophy of the Alibaba Cloud cloud-native observability platform (Cloud Monitor 2.0). Cloud Monitor 2.0 integrates several Alibaba Cloud products—Simple Log Service (SLS), Application Real-Time Monitoring Service (ARMS), Cloud Monitor (CMS), and STAROps—into a single unified platform. It provides full-stack, real-time, and non-intrusive data ingestion capabilities, covering multiple data sources such as logs (hundreds of PB/day), metrics (tens of PB/day), traces (trillions of calls/day), events (billions of records/day), containers, and terminals. Utilizing multi-tier hot and cold storage, it reduces the overall cost by 50% compared to open-source self-built solutions at an exabyte (EB) storage scale. Chanjet's five-layer monitoring data is aggregated, stored, queried, and analyzed through this unified platform, providing a data foundation for upper-layer intelligent applications that supports PB-level daily writes and sub-second analysis of hundreds of billions of data points. The monitoring scope strictly matches the application tiers: Tier-3 applications must cover at least the first three layers, Tier-2 applications extend to the fourth layer, and Tier-1 core applications must achieve full coverage across all five layers. The system adheres to three major principles: comprehensive collection, multi-dimensional validation, and timely delivery. Comprehensive data collection eliminates monitoring blind spots; multi-dimensional cross-validation prevents misjudgments from single metrics; and multi-channel notifications via phone, SMS, DingTalk, and email ensure emergency alerts reach on-call personnel immediately.

Building on this foundation, Chanjet introduced the operations digital twin capabilities of Cloud Monitor 2.0 based on UModel to construct a three-dimensional topology architecture of applications, resources, and tenants. UModel organizes entities, relationships, observability data, and O&M knowledge using a unified graph model. It incorporates cloud products, such as Alibaba Cloud Elastic Compute Service (ECS)/Virtual Private Cloud (VPC)/Server Load Balancer (SLB)/ApsaraDB RDS/Container Service for Kubernetes (ACK), Kubernetes resources (Cluster/Pod/Node/Deployment/Service, etc.), applications (microservices/instances/APIs/HTTP/messages/database calls, etc.), and custom enterprise extensions (CMDB, CI/CD pipelines, self-built middleware, O&M SOPs/knowledge bases, etc.) into a unified semantic model. By combining inter-service call chains and gateway distributed tracing, and linking application and tenant tag profiling, O&M personnel can drill down layer by layer from a single user experience alert to quickly locate anomalous instances, resource bottlenecks, and affected tenants. Simultaneously, the system implements refined alert tiering, aggregation and merging, and escalation mechanisms, consolidating alerts from the same fault source into a single event and relying on the root cause analysis module to assist in troubleshooting. The comprehensive observability data and topology capabilities lay a solid data foundation for subsequent AI-driven end-to-end diagnosis and alert noise reduction, enabling end-to-end analysis to be completed within 30 seconds.

3. AIOps Evolution and Platform Architecture — Efficiently "Managing and Curing"

The observability system solved the problem of issue perception, but moving from fault discovery to complete resolution requires continuous iteration of platform capabilities and O&M models. During its cloud-native transformation, Chanjet divided its AIOps evolution into four stages, with each round of upgrades driving exponential improvements in SLA metrics and comprehensive O&M capabilities.

Stage 1: System Construction Phase (SLA 99.9%).

The core action was establishing application lifecycle management, business tiering models, and foundational red-line standards. Rules were set for release changes—internally compared by Chanjet to "red lines, traffic rules, traffic lights, and cameras." Simultaneously, the multi-center architecture and canary release system were implemented to strengthen disaster recovery and release control capabilities. On the platform side, three foundational centers for monitoring, events, and resources were built to unify standards for data collection, templates, and policies. This stage was dominated by manual operations, with the platform serving merely as a support tool, relying on institutional constraints to reduce human errors.

Stage 2: Platform Empowerment Phase (SLA 99.95%).

Efforts focused on three major dimensions: architecture, historical issues, and the full application lifecycle. The team advanced the cloud-native transformation of all businesses to mitigate systemic risks; conducted targeted governance of historical technical debt, such as strong dependencies and outdated components; and applied differentiated O&M strategies based on application stages. Platform capabilities were comprehensively expanded to form a complete architecture comprising the foundational service layer, platform capability layer, and business layer, providing a vehicle for implementing O&M methodologies.

Stage 3: Methodology Solidification Phase (SLA 99.99%).

With systematic standards in place and the integrated DevSecOps/AIOps foundation built on its self-developed cloud stability platform, Chanjet distilled extensive practical experience into a replicable methodological framework—internally known as the "Yonyou Method." Its core is the "0-2-5-10" emergency response methodology: the fault prevention goal is 0 incidents (proactive); timely perception within 2 minutes (MTTI); root cause analysis within 5 minutes (MTTK); and emergency mitigation and recovery within 10 minutes (MTTF + MTTV). The value of this methodology lies in "solidifying" the best practices accumulated in the first two stages into standardized, executable processes. This ensures that any on-call personnel can achieve consistent response quality by following the framework, elevating the team's overall operational capability from "relying on a few experts" to "executable by everyone."

Stage 4: AI Empowerment Phase (SLA 99.995%).

Building on the dual foundation of platformization and methodology, AI capabilities were comprehensively integrated, fundamentally shifting the operational model from "human-led" to "AI-led, human-verified." The platform interaction layer was upgraded to feature an intelligent cockpit (global view), digital employee (AI virtual employee), and personalized workspaces, delivering AI capabilities in a Model Context Protocol (MCP)/skill mode to achieve continuous delivery and capability reuse of AI. Core AIOps modules include: intelligent inspection (prevention), intelligent alerting (alert noise reduction), intelligent diagnosis (boundary localization), intelligent self-healing (multi-dimensional metric linkage), capacity forecasting, and configuration validation. These modules interlock with the monitoring and event centers to form a closed loop of "perception → analysis → decision → execution." AI large language models (LLMs) are deeply embedded in every stage, with the ultimate vision of completely eliminating manual dependency, achieving proactive fault prevention, minimizing human intervention, and rapidly mitigating losses.

The core logic across these four stages is "shifting from relying on humans to gradually relying on platforms/AI tools." First, build systems to lay the foundation; second, build platforms to serve as vehicles; third, distill methodologies to make them replicable; and finally, superimpose AI to achieve autonomy. Behind every improvement in the SLA lies a leap in capability levels and cognitive understanding.

4. Implementing AI Scenarios — Three Major Closed Loops

Under the AI capability framework of the fourth stage, Chanjet implemented several core AIOps scenarios:

Intelligent Inspection — Prevention ("Physical Examination")

Covering comprehensive O&M risks across five major product lines and nine clusters, the system executes three types of automated inspection tasks. The first is O&M risk inspection, which includes regular scanning across dimensions such as resource capacity trends, configuration compliance, dependency health, and component version risks. The second is database red-line scanning, which uses AI to identify high-risk metrics critical to DBAs, such as slow SQL queries, large table bloat, connection pool levels, missing indexes, and large memory issues. The third is change risk identification, which automatically evaluates the impact scope and risk level before executing changes—including SQL anomaly identification, single-tenant abnormal behavior detection, and resource capacity spike prediction.

Issues discovered during inspections automatically generate work orders that are routed to the responsible teams, forming an automated closed loop of "discovery -> work order -> fix -> verification." This replaces the inefficient legacy model that relied on manual inspections and verbal communication. Based on user-defined inspection goals, the system automatically breaks down task steps and continuously executes asynchronous daily and weekly inspection tasks on a scheduled basis. The data platform based on SLS provides high-performance query capabilities for massive data. Combined with UModel, it rapidly executes inspection query jobs, displaying inspection progress and findings. When high-risk issues are encountered, it automatically triggers manual confirmation to ensure that inspection conclusions are reliable and controllable. Intelligent inspection focuses on "data changes," while intelligent validation focuses on "configuration standards"—the two work in tandem to achieve comprehensive prevention.

Fault Self-Healing — Mitigation ("Medical Treatment")

For identified high-frequency fault scenarios, self-healing strategies are predefined and automatically executed. Typical scenarios include: abnormal resource consumption by a single tenant (triggering user isolation); connection pool levels breaching thresholds (triggering API rate limiting); single node unavailability (triggering center failover); degraded downstream service response (triggering service degradation); and sudden traffic spikes (triggering resource scaling).

Once AI detects the early signs of an anomaly, it automatically matches the best scenario and triggers the corresponding recovery action. Currently, an "AI perception + human verification" model is employed: custom scenario anomalies are identified by AI, which recommends a resolution plan; upon manual verification and confirmation, recovery is executed automatically. During this process, the diagnostic and reasoning capabilities of the AIOps assistant provide core support for locating the root cause of the fault. It automatically gathers evidence around alerts, analyzes the impact scope, correlated services, and anomalous metrics, and uses the UModel operations digital twin to reconstruct the full picture and propagation path of the fault, enabling cross-domain correlation analysis rather than isolated troubleshooting. Self-healing capabilities are hosted by the platform's "AI Recognition/Scheduling Center," achieving unified management of fault injection (verification) and fault self-healing (execution) through job orchestration. Newly added fault patterns can be quickly configured into self-healing rules, reducing the reliance on personal experience for damage mitigation. Meanwhile, manual execution channels are retained to ensure fallback capabilities in extreme scenarios.

Capacity Forecasting and Cost Control — Controlling Costs ("Diet Control")

Replacing the traditional capacity alerting model based on static thresholds, AI time-series forecasting capabilities were introduced. In the traditional model, capacity alerts relied on fixed thresholds (e.g., alerting when CPU > 80%), which suffered from unreasonable threshold settings and an inability to predict future trends. In the new model, AI makes comprehensive judgments based on three dimensions: historical capacity patterns (identifying cyclical peaks and valleys), business growth trends (predicting in correlation with business metrics), and sudden event detection (identifying non-cyclical abnormal growth). Cloud Monitor 2.0 provides rich atomic AI analysis capabilities, including operators for time-series forecasting, time-series clustering, and anomaly detection. These operators push computations down to the underlying engine, completing efficient inference over massive volumes of metric data. The CClaw (Chanjet's Agent platform dedicated for micro and small enterprises) automatically generates daily capacity reports, with AI summarizing trends and issuing risk warnings, predicting capacity shortage risks days in advance. Combined with the FinOps cost control module, it achieves integrated management of "proactive capacity risk prediction → resource lifecycle management → cost utilization optimization → resource scoring and cost optimization." The results are twofold: it reduces costs by avoiding resource waste while simultaneously preventing availability failures caused by insufficient capacity.

5. Outcomes and Value

Through four stages of continuous evolution, Chanjet has achieved a comprehensive breakthrough in core metrics:

The overall SLA improved from 99.9% to 99.995%, experiencing a three-step leap in availability (compressing annual downtime from nearly 9 hours to less than half an hour). Fault localization time was slashed from an average of over 10 minutes to under 30 seconds, boosting the efficiency of the MTTK phase within MTTR by more than 20 times. The emergency response system achieved its set goal of "99% proactive prevention and damage mitigation within 10 minutes"—meaning the vast majority of potential faults are intercepted by intelligent inspection and validation mechanisms before reaching users. The O&M model completed a paradigm shift from "human-led, platform-assisted" to "AI-led, human-verified." On-call duty upgraded from a traditional continuous manual monitoring model to a collaborative model of continuous AI monitoring coupled with human handling of escalated events. This shift freed the O&M team's energy from repetitive fault responses, allowing them to pivot toward higher-value work such as architecture optimization and capability building.

From a deeper capability perspective, the most core change is the establishment of a continuous positive feedback loop: "observability data → AI analysis → automated actions → knowledge accumulation." The experience from every fault resolution is accumulated in the knowledge base and fed back to the AI, continuously improving the coverage and accuracy of intelligent diagnosis and intelligent self-healing. The system becomes "smarter" with use, and its reliance on human intervention steadily decreases.

At the same time, AIOps is no longer just a tool for the O&M team; instead, it guides R&D technical transformations through an experience closed loop of "O&M insights → development standards." Performance bottlenecks and architectural risks discovered by AI are translated into red-line rules and best practices on the R&D side, reducing the probability of faults at the source. For example, a specific slow SQL pattern frequently detected by intelligent diagnosis is automatically extracted into a new entry for database development standards, allowing it to be proactively intercepted during code reviews and release pipelines.

From an organizational collaboration perspective, Chanjet achieved systematic accumulation of O&M knowledge on the observability platform. Troubleshooting experience, architectural understanding, and resolution judgments—previously scattered across the minds of various experts—achieved organizational-level knowledge assetization through the digital employee's knowledge base, skill mechanisms, and MCP extensions. With the help of platform tools and AI assistance, newcomers can quickly get up to speed, significantly improving the team's overall response consistency and reliability.

This closed-loop model—"centered on user experience, driven by AI, and integrated from O&M to R&D"—is precisely Chanjet's core methodology for fully stepping into the AI Agent era and building the next generation of technical operations systems. Serving as an Agentic Ops platform, STAROps of Cloud Monitor 2.0 provides an out-of-the-box technical foundation for implementing this methodology through four core capabilities: unified observability data, operations digital twin, AI analysis operators, and a continuous evolution flywheel.

Bring Ops Capabilities into Qoder: Pinpoint Root Causes in One Sentence

ObservabilityGuy — Mon, 24 Aug 2026 06:19:38 +0000

This article introduces the STAROps plugin for Qoder, letting developers diagnose and fix production issues in natural language.

Almost Every Developer Has Fallen into This Trap

You tweak the business logic of some service. Unit tests all pass, CI is green, the code review is approved—a smooth launch to production. Ten minutes later the monitoring alarms explode: the service's response time shoots straight up.

You go over the diff twice, and the logic looks flawless. But to pin down the root cause, you have to hop across at least five platforms: search logs in SLS, stitch together metrics in Grafana, check traces in APM, dig through the release system for changes, and ask the ops team for CMDB topology data. Every platform has its own query syntax, permissions block you halfway, and in the end you still have to @ an SRE in the group chat to pull data for you. After several rounds of back-and-forth cross-team coordination, half an hour is gone, and all you could do the whole time was stare at the chat window waiting for a reply.

And this kind of scene probably plays out in your engineering team every single week.

The real problem was never a lack of tools. According to Gartner's 2025 DevOps toolchain report, mid-to-large enterprises deploy an average of 6–8 ops and monitoring tools, spanning monitoring, logging, tracing, change management, incident management, and more—nothing is missing. But these tools are built for SREs and ops teams. Their core design goals are "comprehensive, professional, and customizable," which translates into complex query syntax, specialized concept systems, and long operational paths.

The heart of the contradiction is a mismatch between the tools and their target users. For developers, production troubleshooting is a low-frequency emergency scenario. Spending an hour learning PromQL or SLS query syntax just to handle one incident has a far worse return on investment than simply asking ops for help—which is exactly what creates cross-team communication overhead and traps ops teams in a flood of repetitive data-pulling chores, leaving them no time for the more essential work of building reliability. What developers want has never been to learn ops tools; it's to get actionable diagnostic conclusions directly. This isn't about replacing ops—it's about pushing standardized diagnostic capabilities down to the development side, so both sides can focus on their own core value.

What if there were a way to query production, view diagnostics, and ask about root causes—all without ever leaving your AI coding tool?

When STAROps Lives inside Qoder

First, a one-sentence introduction to Alibaba Cloud STAROps, the full-domain intelligent ops platform: query metrics, analyze logs, trace calls, and diagnose alerts—all in natural language. Behind it is UModel, the unified ops data model Alibaba Cloud has refined over many years. Unlike a traditional CMDB that only records static asset relationships, UModel breaks down the data silos between different ops tools and builds a full-element semantic network of applications, services, resources, alerts, and changes, unifying entity relationships and data definitions. This is the core foundation that lets a large model perform accurate cross-domain root-cause reasoning, eliminating at the source the problems of misaligned data and wrong correlations across tools.

STAROps is already powerful on its own—ops teams handle their day-to-day diagnostics and inspections through its console or IM. Now this capability extends further, into developers' AI coding tools. Once you install the official STAROps plugin, you ask questions in natural language right inside Qoder's chat box, STAROps performs the cross-domain data queries and root-cause reasoning, and the structured conclusion appears directly in your Qoder. No switching windows. No waiting on ops colleagues. No learning any new query syntax. This means developers, for the first time, have visual diagnostic capability for the production environment—while you write code, you can glance at the real state of production anytime, and not by flipping through monitoring dashboards, but as naturally as chatting with a colleague.

Three Scenarios: See How It Actually Works

The three scenarios below are arranged along the timeline of everyday development: how to investigate when something breaks, how to drill deeper after investigating, and how to check before you make a change.

Scenario 1: A Service Throws an Error—Just Ask Right in the IDE

Back to the example from the start. After the release, response times spiked, and now you just ask right inside Qoder:

I'm getting a high-latency alert on product-catalog ListProducts---P95 jumped from <60ms to 1875ms. Analyze the root cause.

As soon as STAROps receives the request, it gets to work. It does a few things: first it checks the service's recent error logs, extracting exception stacks and error patterns; then it pulls APM metrics to look at trends in P95 latency, container replica count, and throughput; next it examines the topology to see whether the call chains of upstream and downstream services show anything abnormal; and finally it correlates change records—pulling up the list of recent release events and comparing each one's deployment time against the latency curve. The entire analysis streams back, so right in the Qoder chat box you can watch STAROps gather evidence and reason toward the root cause step by step. The final conclusion might look like this:

Root cause analysis: DB connection pool starvation (MaxOpenConns=1, MaxIdleConns=1). Triggering version: v2.2.0-buggy (commit d9420f7, ticket OPS-1024, operator David Zhang). Evidence chain: After v2.2.0-buggy was deployed at 14:06, P95 latency immediately surged from <60ms to 1875.8ms, and container ReadyReplicas jumped from 2 to 24 (auto-scaling triggered), a deviation exceeding ±4.1σ. Latency recovered immediately after the rollback to v2.1.0. Core mechanism: ListProducts needs to run multiple GetProduct calls concurrently → multiple SELECTs contend for a single database connection at once → large numbers of requests queue on the connection pool, and the actual time of a single SELECT is amplified by the queue wait to 300ms ~ 3400ms. Conclusion: The v2.2.0-buggy version lowered MaxOpenConns from a reasonable value to 1, so concurrent GetProduct calls queue on the database connection pool, driving ListProducts P95 latency from <60ms up to 1875.8ms. We recommend rolling back to v2.1.0 immediately and isolating the problem version behind a feature flag. Confidence: 80%.

In the traditional model, this kind of post-release troubleshooting requires crossing more than three platforms—container monitoring for replica counts and resource metrics, the APM platform for call chains and latency distribution, the release system for deployment records—coordinating two ops colleagues and taking more than 40 minutes on average. With Qoder + STAROps, going from question to conclusion might take just two or three minutes. And this conclusion isn't a "go dig through the logs yourself"—it has already cross-correlated container metrics, latency curves, the release timeline, and configuration differences for you. It's a reasoned diagnostic conclusion, and you can change your code based on it directly.

Scenario 2: Diagnosis Isn't a One-Shot Deal

Real-world troubleshooting is rarely pinned down in a single round. You have the preliminary conclusion of "connection pool starvation," but you still need to confirm more: did the latency spike start the very moment v2.2.0-buggy was deployed, or was there a gradual buildup? Was it caused by the single ListProducts endpoint, or was it system-wide? What exactly did v2.2.0-buggy change in its configuration? You need to drill down further to verify—and all of this, again, without leaving the IDE and without re-entering the context.

In Qoder, you just follow up:

Overlay and compare the release times with the latency curve, confirm the causality.
Compare the key configuration differences between v2.2.0-buggy and v2.1.0. Which endpoints are affected the most?
Is there any anomaly in connection release latency?

STAROps supports multi-turn conversation. The context is preserved throughout the same conversation thread—it knows you're still asking about the product-catalog service, it knows you care about the connection pool problem tied to ticket OPS-1024, and it won't rescan all the data on every round. Just like chatting with an SRE colleague who knows the system well, you ask follow-up questions round by round, gradually narrowing the scope of the investigation.

It pulls up a correlation analysis of release times and the latency curve—latency rose immediately after v2.2.0-buggy was deployed at 14:06, P95 climbed sharply to its peak during the second deployment between 14:06 and 14:22, and latency returned to normal after the rollback to v2.1.0. The timing lines up precisely. It helps you compare configuration differences between versions—pinpointing the specific changes to connection-pool parameters like SetMaxOpenConns, SetMaxIdleConns, and SetConnMaxLifetime, as well as the abnormal introduction of a SetProduct write operation. In the end you might pin it down to this: v2.2.0-buggy not only squeezed the connection pool to its limit, it also inserted an unnecessary database write into the query path, and the two factors combined drained the connection pool instantly.

The root cause is found, and the fix is clear too—restore the connection-pool parameters to reasonable values, remove the redundant SetProduct write, and add a 100ms context timeout to the query to keep slow queries from blocking.

The value of this scenario is depth. A single round of diagnosis gives you a direction; multiple rounds of follow-up help you pin the problem down to the specific code change. The whole process requires no ops query syntax at all—you don't need to understand Prometheus QL or SLS query syntax; you only need to describe what you want to know in natural language.

Scenario 3: The Root Cause Is Found—Now How to Fix It

The first two scenarios helped you locate the root cause: the v2.2.0-buggy version of the product-catalog service squeezed the database connection pool to its limit and introduced an unnecessary write operation into the query path, driving the ListProducts endpoint's P95 latency up to 1875.8ms. But diagnosis isn't the finish line—you still need to turn this conclusion into a concrete code fix, get it committed, and push it toward release.

Keep asking in Qoder:

How should the OPS-1024 connection pool starvation problem be fixed? Give me a concrete code fix, and once it's done, submit an MR for me.

Qoder + STAROps doesn't just tell you "where the problem is"—it can fix it directly for you. Based on the earlier diagnostic context—ticket OPS-1024, the connection-pool parameter changes introduced by buggy commit d9420f7, and the ListProducts endpoint's P95 jumping from <60ms to 1875.8ms—it generates concrete fix code:

Fix plan (12 changes): File to fix: src/product-catalog/main.go

What you get isn't a vague "check your connection pool configuration," but a fix precise down to the code file and the specific parameters. But what matters more is what happens next—you don't have to perform these changes by hand. Qoder + STAROps takes over the entire commit process: it automatically creates the fix branch fix/product-catalog/revert-ops-1024-pool, commits & pushes the modified code to the remote. Then, through the MCP protocol, it calls the Yunxiao Codeup API to automatically create a MergeRequest targeting the master branch, with the MR title fix/product-catalog: revert OPS-1024 DB pool starvation and pg_sleep audit—even the MR description is auto-generated, including the full incident background, root-cause analysis, and fix notes. When you open Yunxiao Codeup in your browser, the MR is already there waiting for your review.

The value of this scenario is the closed loop. In the traditional workflow, there's still a "translation cost" between locating the root cause and writing the fix code—you have to understand the technical details of the problem yourself, figure out how to change it and which files to touch, and then manually go through the Git flow and log in to the code platform to create an MR. Qoder + STAROps eliminates that entire cost: the diagnostic conclusion connects directly to the fix code, and the fix code turns directly into a MergeRequest. From discovering the problem to an MR waiting for review, the whole process can be done in a single IDE window, with no need to switch to any other platform in between. A developer's coding decisions are no longer based only on code logic and local tests—they're backed by real production data. The fix you submit isn't just "logically correct," it's also "aware of the production environment."

What Happens behind the Scenes

Now that you've seen the three scenarios, you might be wondering: how is all this done?

The answer is Qoder's plugin mechanism. STAROps provides an official plugin; once you install it with one click from the Qoder plugin marketplace, every ops-related question you ask in the chat box is routed to STAROps.

The call chain is simple: you type natural language → Qoder recognizes the ops intent → the request is forwarded to STAROps → STAROps performs cross-domain data queries and root-cause reasoning → the structured conclusion returns to your IDE.

The security mechanism follows Alibaba Cloud's enterprise-grade standards: it inherits RAM permissions (no privilege escalation), performs read-only queries (no changes), applies automatic data masking (no leaks), and keeps a full audit trail (traceable). Credentials use the default Credentials SDK chain and support environment variables, config files, and OIDC—no plaintext keys required.

It's worth noting that STAROps comes in three capability forms: intelligent assistant (instant Q&A diagnosis), long-running task (continuous inspection and guarding), and digital employee (an ops agent with configurable duties and permissions). What you invoke inside Qoder is the intelligent assistant—instant, precise, and triggered on demand, best suited for developers to quickly gain ops insights while coding. If you later need continuous monitoring (for example, "automatically keep an eye on things for an hour after a release"), you can upgrade to a long-running task.

How to Get Started

Three steps to get going, 3 minutes end to end:

Step 1: Install the STAROps plugin. In Qoder Desktop, switch to the Quest view and search for "STAROps" in the plugin marketplace to install it with one click.

Step 2: Configure your Alibaba Cloud credentials. It follows the default Credentials SDK chain standard and supports multiple methods—environment variables, config files, OIDC, and more—with no need to configure plaintext keys.

Step 3: Start asking. Open the Qoder chat box and simply describe, in natural language, the ops question you want to investigate.

New STAROps users get 10,000 credits valid for one month, plus an additional free allowance of 2,500 credits each month. For reference: a single lightweight query costs about 30 credits, and a full cross-domain root-cause diagnosis costs about 200 credits.

Shifting Ops Left: A Trend Already Underway

At this point, what this article describes is really one concrete thing: developers gain STAROps's ops diagnostic capabilities through Qoder. But if you pull back a little, you'll find the significance goes beyond "you can query production from the IDE."

In the traditional model, the capability boundary between development and ops is rigid. Developers write code, ops keeps the system running, and the two rely on people relaying messages, tickets circulating, and meetings to stay in sync. After the STAROps plugin connects to Qoder, this boundary is crossed by technology rather than people for the first time—developers can gain ops insights without learning ops tools, and ops teams no longer have to pull logs for developers, because diagnostic capability becomes infrastructure available to everyone through Qoder and the STAROps plugin.

This is the first step toward breaking down the information barrier between Dev and Ops. For developers, there's no more playing telephone with information—you can sense the production state as you write code, and troubleshooting compresses from hours to minutes. For ops teams, it dramatically reduces the energy consumed by repetitive data pulls and basic troubleshooting tickets, freeing up time to focus on high-value work like architecture optimization and building reliability systems. When development and ops share the same production context, not only will incident recovery be faster, but those "hit the same trap over and over" problems will grow rarer—ultimately delivering a two-way boost to the efficiency and stability of the entire engineering team.

You can head straight to qoder.com to download Qoder, finish setup in 3 minutes, and immediately experience troubleshooting production issues with a single sentence inside the IDE. Register now to claim 10,000 STAROps credits and put production-environment diagnostic capability right into your Qoder.

Zero-Code Instrumentation: See Through Every AI Agent Invocation

ObservabilityGuy — Mon, 24 Aug 2026 06:11:37 +0000

This article introduces OBI, a zero-code, kernel-level solution that automatically captures full AI agent invocation chains and emits GenAI-compliant OpenTelemetry traces and metrics.

When an AI agent receives a user query, it first calls an embedding model to vectorize the query, then initiates a Top-K retrieval to Pinecone, followed by a reranking process. After that, it calls GPT-4o with the context. During this time, GPT-4o might also invoke external tools via the MCP protocol. This entire chain spans five protocols and three cloud providers. When a user complains that "the answer is wrong," developers are faced with a crime scene but have no surveillance footage to investigate.

Traditional APM can tell you "an HTTP request took 3 seconds," but it cannot answer "which model was used, how many tokens were consumed, what functions the tool call invoked, or how many results the vector search returned."

The approach of OpenTelemetry eBPF Instrumentation (OBI), on the other hand, is to install a 24/7 forensic camera inside the Linux kernel. Without modifying a single line of business code, it automatically identifies and parses all AI-related network calls, recording the key evidence into OpenTelemetry standard traces and metrics.

Why Is Manual Instrumentation Based on GenAI Semantic Conventions So Hard?

The OpenTelemetry community has defined a set of semantic conventions for GenAI, specifying standard attributes like gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. Ideally, all AI applications should report data according to this specification so that calls from different providers can be monitored on a unified dashboard and covered by the same alerting rules.

In reality, however, manually instrumenting business code to meet these standards is an agonizing process:

SDK fragmentation: Every provider's SDK is entirely different. The OpenAI SDK, Anthropic SDK, Google GenAI SDK, Boto3 (Bedrock), and DashScope SDK represent five different API wrappers. Developers must write tracing wrappers for each, extract their respective model, token, and tool_calls fields, and map them to unified GenAI attributes.

Rapid evolution of semantics: The GenAI semantic conventions are still evolving quickly. They only moved from an experimental state to stable in 2024, and field names and enum values are constantly being tweaked. SDK wrapper maintainers have to constantly play catch-up, and data generated by older versions might not be compatible with newer ones.

Multi-language complexity: Adapting for multiple languages is a multiplicative problem. Python's opentelemetry-instrumentation-openai and Go's community wrappers are completely separate projects with different maintainers and varying levels of maturity. Observability support for the same provider can be highly inconsistent across different language ecosystems.

Intrusive modifications: Code changes are unavoidable. You have to modify code -> install packages -> align versions -> retest -> redeploy. Integrating a new AI service becomes a full-blown engineering project, severely slowing down iteration speed.

There is an even more fundamental issue: many AI agents do not even use official SDKs. A lot of frameworks and in-house apps simply use standard HTTP clients like requests, http.Client, or fetch to construct JSON request bodies and call LLM APIs directly. It's lightweight, flexible, and avoids SDK version lock-in. But this also means that all observability solutions based on SDK monkey-patching or wrappers completely fail. There are no SDK objects to hook into, no callbacks to inject, and instrumentation libraries have nowhere to start. For OBI, however, whether you use an official SDK or raw HTTP requests, it all ultimately boils down to HTTP traffic over TCP. What the kernel sees is exactly the same.

The result is that a massive number of AI applications are stuck in an "observability blind spot." It's not that developers don't want to monitor them; it's just that the cost of adopting SDK-based solutions is too high, and raw HTTP scenarios are completely unsupportable by those traditional means.

OpenTelemetry's Non-Intrusive Solution: Pushing Observability Down to the Kernel

OBI approaches the problem from a different level: instead of wrapping SDKs provider by provider at the application layer, it uniformly intercepts HTTP traffic at the network layer of the Linux kernel. Through protocol-level parsing, it automatically extracts all the fields required by the GenAI semantic conventions.

This means one set of probes covers all providers. When the specs update, you only need to upgrade the OBI DaemonSet, requiring zero changes to your application. It is inherently cross-language—whether it's Python, Go, Java, or Node.js, they all send HTTP requests over TCP, and the kernel does not care what language the app uses. More importantly, whether the app uses official SDKs or raw HTTP requests, OBI captures them equally. It doesn't look for SDK objects; it looks at the actual HTTP requests and responses flowing through the network.

The following diagram illustrates how OBI automatically sets collection points at every hop in a typical AI agent invocation chain. From embedding, vector retrieval, and reranking to LLM inference and MCP tool calls, all outbound HTTP requests are transparently captured at the kernel layer:

When OBI captures an AI-related HTTP request, it goes through the following processing pipeline—from packet reception at the NIC to finally outputting an OTel Span that complies with GenAI semantic conventions:

How Does the Kernel "See" an Encrypted LLM Call?

Pushing observability to the kernel sounds great, but it immediately hits a wall: today, all LLM calls run over HTTPS. If you capture packets directly at the NIC or socket layer, all you see is a bunch of encrypted bytes. You can't parse the model or tool_calls, let alone reconstruct an SSE stream. For OBI to gather evidence at the kernel layer, it first had to solve a fundamental problem: how to get plaintext from HTTPS without decrypting private keys.

Step 1: Place the probe at the exact line of code before TLS encryption. OBI doesn't touch the ciphertext at the TCP layer. Instead, it hooks (via uprobe) into user-space cryptographic libraries at the exact moment of "pre-encryption/post-decryption." Specifically, it adapts to four types of runtimes:

OpenSSL/BoringSSL dynamic libraries: It attaches uprobes and uretprobes to SSL_write and SSL_read in libssl.so. The entry of SSL_write yields the plaintext buffer the application just handed over for encryption, and the return of SSL_read yields the freshly decrypted plaintext buffer. Node.js, CPython, curl, and nginx all use this path.
Go statically linked binaries: Go has its own crypto/tls and doesn't rely on the system's libssl. Furthermore, symbols might be inlined and renamed. OBI uses ELF symbol tables and DWARF information to locate the function offsets for crypto/tls.(*Conn).Write and Read, attaching uprobes directly by offset to statically linked binaries.
Python _ssl extension: CPython's ssl module calls _ssl.so under the hood, which ultimately falls back to OpenSSL, reusing the first path.
Stripped binaries fallback: When symbols are stripped, OBI uses BPF Type Format (BTF) and .eh_frame stack unwinding information to locate critical function entries. This ensures it still works on production images where debugging symbols were removed during compilation.

The key insight here is that encryption happens inside the user-space crypto library. The interface between the application layer and the crypto library is always a plaintext buffer. By hooking this interface, you can get the complete request and response bodies without needing man-in-the-middle certificates or private keys.

Step 2: Move data from kernel to user space with zero-copy. After the uprobe is triggered, the eBPF program needs to pass the captured HTTP plaintext to the user-space OBI agent for protocol parsing. The traditional perf event mechanism requires multiple buffer copies per event, which is an unacceptably high cost for LLM prompts that can easily span tens of kilobytes. OBI uses the BPF ringbuf (introduced in Linux 5.8): the kernel eBPF program allocates space directly in shared memory via bpf_ringbuf_reserve, and writes application data via bpf_probe_read_user in one go. The user-space reader maps the same memory segment via mmap for zero-copy reading. Backpressure is managed by the ringbuf's own watermarks; when the high-water mark is hit, the kernel side drops events and logs metrics, ensuring the business process is unaffected. In multi-CPU scenarios, each CPU gets its own ringbuf slice to avoid lock contention.

Step 3: From HTTP plaintext stream to OTel Span. Once the plaintext enters user space, it goes through a complete parsing pipeline. This involves HTTP/1.1 and HTTP/2 frame reassembly, selecting JSON/SSE/Binary decoders based on the content type (Content-Type), accumulating streaming responses by event, extracting fields mapped to GenAI semantic conventions, enriching them with K8s metadata (pod, namespace, service, workload), and finally outputting an OTLP Span. This entire pipeline runs on the OBI agent's swarm DAG scheduler. Every stage is a horizontally scalable actor, keeping CPU usage stably under 1% for typical single-node workloads.

The following diagram connects these three steps into a complete data path—from the application sending a request, to the uprobe capturing the plaintext, zero-copying via ringbuf, and finally parsing the protocol to output an OTel Span:

Cross-Language Coroutine Tracing: Why PID Isn't Enough

Capturing plaintext is only a data-layer victory; the harder part is the correlation layer. A real-world AI agent is almost never a synchronous "one request per thread" model. Python asyncio runs dozens of coroutines concurrently on a single thread, Go uses goroutines to constantly switch context, and Node.js scatters callback chains everywhere. If you simply grouped all calls under the same parent span using traditional PID/TID, the entire trace would be completely tangled.

OBI rebuilds context at the kernel layer for the three mainstream concurrency models. Let's take Python and Go as examples.

Python asyncio: Reconstructing parent-child relationships with 4 uprobes. CPython's asyncio event loop is a classic example of single-threaded multitasking—all coroutines run on the same OS thread and switch via Task.step(). OBI attaches 4 uprobes to the CPython interpreter, monitoring four critical points in a coroutine's lifecycle:

uprobe Hook Point	Timing	Extracted Information
task_step	Coroutine scheduled to execute	Extracts the current Task object pointer, used as the coroutine ID.
Task.init	New coroutine created	Extracts parent-child lineage by recording the Task running when this new Task was created.
PyContext_CopyCurrent	Context copied	Takes a snapshot of contextvars, used as a data channel between coroutines.
context_run	Callback executed in specific context	Extracts the currently active context, linking to the correct coroutine.

By coordinating these four points, OBI maintains a mapping table in the kernel: coroutine ID -> parent coroutine ID -> current trace context. Even if 10 coroutines are concurrently calling LLMs in a single thread, OBI accurately determines which coroutine initiated each HTTP request and which trace it belongs to.

Go goroutine: Extracting lineage from within the runtime. Go's goroutines are even harder to trace than asyncio. Scheduling is handled by the runtime, it doesn't expose any stable user-space APIs, and even the goroutine ID is intentionally hidden. OBI goes directly for internal Go runtime functions. runtime.newproc1 is the entry point where a parent goroutine forks a child. OBI records the Parent G pointer -> Child G pointer here, establishing a lineage table. runtime.casgstatus handles goroutine state machine switching, which OBI uses to detect when a G is bound to an M (OS thread) or preempted. When an outbound HTTP request triggers, find_parent_goroutine traces up the lineage table for up to 6 levels to find the nearest ancestor goroutine with a trace context—this is the key to rebuilding the Go coroutine chain.

Why 6 levels? The OBI team analyzed real Go applications and found that 6 levels cover 99% of goroutine creation depths. Going deeper usually hits internal framework worker pools, which actually blurs the business context.

Cross-process tracing: Kernel tpinjector injects traceparent. After correlating coroutines within an application, cross-service chaining must be addressed. OBI uses bpf_probe_write_user directly in the kernel on the header section of outbound HTTP requests, injecting a traceparent header into the plaintext buffer right before SSL_write encrypts it. When the downstream service receives it, it goes through a symmetrical decryption process. OBI captures this header at the SSL_read exit, thereby stitching the entire trace across processes, services, and languages. This entire process is completely transparent to the application; even the HTTP client code is unaware that a header was added to its outgoing request.

Performance overhead: How do we achieve <1% CPU usage? This set of mechanisms sounds heavy, but OBI's typical overhead on production clusters is stably under 1% CPU. There are three key reasons: First, uprobe trigger frequency is limited by actual HTTP call rates; unlike kprobes, it won't get overwhelmed by high-frequency system calls. Second, BPF ringbuf batching means the user-space reader wakes up once to consume multiple events, avoiding per-event context switches. Third, protocol parsing happens in user space, not the kernel. The kernel eBPF only does the thinnest "capture buffer + insert into ringbuf" work, offloading complex field extraction to user-space actors, avoiding BPF verifier complexity explosion.

The following diagram illustrates a scenario where 4 concurrent LLM calls run on a single Python asyncio thread. A traditional PID/TID correlation would incorrectly group them all under one parent span. By reconstructing the coroutine context via 4 uprobes, OBI correctly restores the true parent-child relationships of 4 independent traces:

Protocol Parsing State Machine: From Byte Stream to GenAI Span

Once the plaintext enters user space, OBI faces an unlabeled stream of HTTP bytes. It needs to determine in milliseconds whether it's OpenAI or Anthropic, Chat or Embedding, RAG retrieval or an MCP tool call, and then extract the GenAI fields according to their respective specs. This decision logic isn't a simple if-else; it's a three-stage state machine. Relying on any single stage alone leads to false positives, but combining all three ensures absolute precision.

Stage 1: response header signatures (highest priority). Every LLM provider leaves unique fingerprints in their response headers:

Provider	Response Header Fields
OpenAI	Openai-Version, Openai-Organization
Anthropic	Anthropic-Organization-Id, Anthropic-Ratelimit-*
Gemini	X-Gemini-Service-Tier
Qwen	X-DashScope-Request-Id
Bedrock	X-Amzn-Bedrock-Input-Token-Count (tokens directly in the header)

Why are response headers the most reliable? Because they are added by the provider and cannot be forged by the application layer. However, this method fails during 4xx/5xx error responses, where many providers switch to a generic error path that lacks these custom headers.

Stage 2: URL host + path two-step verification (fallback). When response headers are missing, OBI falls back to checking the request URL. For example, dashscope.aliyuncs.com + /chat/completions is Qwen, bedrock-runtime.amazonaws.com is Bedrock, and generativelanguage.googleapis.com + /models/ is Gemini. This stage covers scenarios with incomplete response headers, such as error responses or interrupted streaming responses. But looking at the URL alone can be deceiving: many companies use an internal LLM gateway (for unified auth, billing, and rate limiting). All apps might call internal-llm.example.com/v1/chat/completions. The URL looks OpenAI-compatible, but the backend could route to any provider.

Stage 3: request/response body top-level key verification (final verdict). OBI performs a final body check on the identified requests. An LLM call must have the model top-level key + messages or prompt fields. Embedding must have model + input. Rerank must have model + query + documents (matching 2 out of 3 is enough, accommodating slight differences between Cohere, Jina, Voyage, and Qwen). Vector search must hit at least two feature keys (e.g., vector + topK, namespace + includeMetadata) from the key sets of six major vector databases (Pinecone, Qdrant, Milvus, Zilliz, Chroma, Weaviate) to prevent normal KV queries from being misidentified. MCP tool calls require a JSON-RPC 2.0 structure (jsonrpc:"2.0" + method + id) + an MCP method whitelist (tools/call, resources/read, prompts/get, etc.) + an Mcp-Session-Id header; all three layers are indispensable.

By coordinating this three-stage state machine, OBI can accurately identify standard calls with normal responses, error responses, and even internal LLM gateway routing scenarios.

SSE streaming responses: "watching" a conversation inside the kernel. Streaming responses are the trickiest part of LLM interactions. A complete conversation is broken down into dozens or hundreds of SSE events, each carrying just one token fragment. OBI maintains an accumulation buffer in user space indexed by trace ID, rebuilding it event by event. Taking Anthropic streaming as an example: message_start creates the session context, content_block_start opens a content block, content_block_delta appends tokens, message_delta carries the final usage, and finally, it outputs an OTel Span that is fully equivalent to a non-streaming API call. This is why OBI can accurately calculate input/output tokens in streaming scenarios—it doesn't "guess" at the end of the request; it accumulates them in real-time, event by event.

MCP tracking. MCP is the new standard for AI agents invoking external tools. The traffic is standard HTTP + JSON-RPC 2.0. OBI accurately identifies MCP calls via a three-layer disambiguation: the Mcp-Session-Id header + method whitelist + protocolVersion. Once confirmed, it extracts semantic information based on the method type: tools/call extracts the tool name, parameters, and return values; resources/read extracts the resource URI; prompts/get extracts the template name. All this information is linked to the same trace via the session ID.

Integrating Monitoring via Cloud Monitor 2.0

AI Agent Integration

You can integrate your AI agents with a single click via the Cloud Monitor 2.0 Integration Center [1]. Once integrated, you can locate your application in the AI Agent Observability dashboard and view AI-related monitoring data:

Real-World Scenarios: Three Troubleshooting Stories with OBI

Collecting comprehensive data is only the first step. The key is whether you can pinpoint issues before users complain. The following three scenarios are typical cases collected during actual deployments, corresponding to the three types of issues OBI is most often used to "crack": recall quality, token costs, and blind spots in proprietary, self-developed apps.

Scenario 1: "Irrelevant Answers"—Is It Bad Retrieval or a Bad Model?

A week after launching a document QA agent, users reported it was "often making things up." Developers checked APM logs and only saw a /chat/completions call taking 2.8 seconds with a status code 200. Beyond that, there were no clues—the model, vector database, and reranking were all black boxes.

OBI's RAG analysis view unfolded the entire trace: the embedding span showed the query used text-embedding-3-small. The subsequent Pinecone span revealed Top-K=5, namespace=docs-v2, hits=1, and highest score=0.31. The reranking span showed the order remained unchanged. Connecting these three pieces of data instantly pinpointed the problem: it wasn't a model hallucination. The vector database namespace was misspelled; the new version of the documents wasn't pointing to this index. The entire troubleshooting process took less than a minute. In the past, relying on added logs and binary searches would have taken at least half a day.

Scenario 2: Token Bills Surged 3x at the Start of the Month—Who's the Big Spender?

A business unit received their Bedrock bill at the beginning of the month: input tokens had spiked by 320% year-over-year. Developers faced dozens of microservices and several agents, and no one could say which piece of code was burning cash. Traditional APM doesn't expose token fields at all, and even if SDKs were instrumented, the data would be scattered across application logs and impossible to aggregate.

OBI aggregated all LLM calls on a dashboard by gen_ai.request.model + service.name + gen_ai.usage.input_tokens. In 30 seconds, the anomaly was locked down: an internal knowledge base agent averaged 80,000 input tokens per call—40 times higher than other apps in the cluster. Drilling down into the model call details to view the specific prompt revealed that a developer, trying to "improve accuracy," was stuffing entire PDFs directly into the system message, sending them repeatedly with every conversation turn. OBI caught this detail at the kernel layer, requiring zero cooperation from the business team to change code or add instrumentation.

Scenario 3: Where Self-Developed Agents Building Raw HTTP Requests Make SDK Probes Completely Useless

A team built a custom agent using Python requests to call Qwen's /chat/completions directly, bypassing any official SDK so they could control retry and timeout logic themselves. Under this path, all OpenTelemetry instrumentation libraries based on SDK monkey-patching stopped working entirely. The team assumed their custom agent couldn't be monitored.

After deploying OBI, without changing a single line of code or installing any Python packages, the invocation chain immediately appeared on the AI Agent Observability dashboard. The provider was auto-identified as Qwen, and model, input/output tokens, and tool_calls were all fully populated, looking completely identical to an app using the official SDK. The reason was explained earlier: OBI looks at HTTP packets flowing over TCP and doesn't care if the app uses an SDK. This is OBI's most hardcore differentiator compared to any SDK solution: its coverage isn't "applications that integrated our SDK," but "all applications that call LLMs via HTTP."

The two screenshots below are from the same OBI AI Agent Observability dashboard. One is the raw-http-agent from Scenario 3 using the standard library http.client to build requests. The other is an openai-mcp-demo using the official OpenAI SDK. The fields under the "Model Analysis" view—call volume, average latency, total tokens, model dimensions (qwen-plus/text-embedding-v3), and the distribution of latency and volume per model—are structurally identical.

raw-http-agent	openai-mcp-demo

Future Plans

The current version of OBI has already achieved comprehensive tracking of GenAI invocation chains, but this is just the beginning. Next, we will focus on advancing the following areas:

Time to First Token (TTFT): For streaming response scenarios, the latency from request to the arrival of the first SSE event is the most direct metric for user experience. OBI will accurately record this time delta at the kernel layer, helping developers pinpoint whether a "slow model response" is due to the network, queuing, or the inference itself.

GenAI-specific metrics: Beyond traces, OBI will generate a set of metrics tailored for AI scenarios. This includes token consumption rates, success/error rates aggregated by provider and model, average response latency percentiles, and Top-N tool call frequencies. These metrics can be directly plugged into Prometheus/Grafana or Alibaba Cloud ARMS, delivering an out-of-the-box AI app monitoring dashboard. We will also continuously track API signatures of emerging AI providers (like DeepSeek and Mistral) to ensure provider recognition stays ahead of users' needs.

End-to-end agent observability: As AI applications evolve from single-turn QA to multi-step agent architectures, OBI will provide end-to-end tracking capabilities for agent execution chains. This includes automatically identifying the agent's planning -> tool call -> observation -> response loop at the kernel layer, chaining each round of decision-making and tool usage into a complete agent trace. It will support MCP semantic recognition, automatically tagging tool names, parameter summaries, and return statuses. For mainstream agent patterns like ReAct and Function Calling, it will offer dedicated invocation topology views and latency waterfall charts, helping developers pinpoint "which step the agent is stuck on" or "which tool slowed down the overall response."

Multi-turn conversation context correlation: Chaining multiple LLM calls within the same session into a complete conversational flow, supporting the analysis of token consumption trends and response quality degradation at the conversation dimension.

Conclusion

Returning to the "crime scene with no surveillance footage" from the beginning: when users complain about "wrong answers," developers no longer need to guess, simulate, or write custom instrumentation wrappers for every provider. As long as it runs on Linux, OBI can automatically capture every LLM call, every tool call, and every vector search at the kernel layer, outputting standard telemetry data that complies with GenAI semantic conventions. It's not about burying probes inside your application; it's about turning the operating system into a holographic flight recorder for your AI agents. Whether you use an official SDK or a raw HTTP request, whether it's Python, Go, Java, or Node.js—they all submit the same data format and flow into the same dashboards. Semantic convention updates only require a DaemonSet upgrade with zero app modifications. For teams moving toward multi-step agents, multi-model orchestration, and hybrid provider deployments, this means observability is no longer an afterthought patched on post-launch, but foundational infrastructure present from the very first packet.

Why Is Your AI Agent Slow? Node.js Agent Connects Models, Tools, and Service Traces in One Go

ObservabilityGuy — Mon, 24 Aug 2026 05:44:35 +0000

This article introduces the ARMS Node.js agent, unifying tracing, runtime metrics, logs, and AI observability in a single integration.

Introduction: The Problem Isn't a Lack of Server-Side Monitoring, but Cross-Layer Complexity

Today's development teams are not short on server-side monitoring.

You probably already have logging platforms, basic metrics, and APM tools in place. The real headache isn't a lack of data; it's whether you can put all this data into the same context when an issue arises.

For example, a user complains: "The AI assistant took forever to answer this time."

You check the entry API and find the response time is indeed high. You check the database—no slow SQL queries. You check Redis—the hit rate is normal. You sift through logs—no errors. Digging deeper, the root cause might be hiding in a LangChain tool call, a sudden spike in the model's time-to-first-token (TTFT), or perhaps the Node.js event loop was blocked for 200 ms by a piece of synchronous logic.

Server-side monitoring is common, but modern Node.js services do much more than just "receive requests, query databases, and return JSON." They increasingly serve as Backend for Frontend (BFF), API gateways, real-time communication hubs, queue consumers, and AI Agent orchestration layers. A single request might simultaneously traverse HTTP, databases, caches, RPCs, message queues, runtime resources, and LLMs.

Therefore, this article won't debate whether you need server-side monitoring. The answer is an obvious yes.

What we really want to discuss is this: When a Node.js application becomes the convergence point for business traffic, asynchronous orchestration, and AI invocations, how can you use a single agent to integrate entry requests, dependency calls, runtime states, log contexts, and AI invocations into a unified troubleshooting trace?

Why Do We Need Another Node.js Agent?

It's not that server-side monitoring doesn't exist, but rather that the problems teams need to solve have changed.

First, Node.js is becoming a "trace convergence layer," and issues are no longer confined to a single API.

Many enterprises use Node.js for BFFs, API gateways, frontend-backend adaptation layers, and AI service orchestration. While it might not be the heaviest business system, it often stands directly between user experience and backend dependencies. If the entry point slows down, users immediately perceive the Node.js service as slow, yet the root cause might lie in the database, cache, downstream RPC, message queue, or model invocation.

Making matters more complex, Node.js is inherently built around Promises, async/await, timers, callbacks, and the event loop. After a user request enters the service, it might cross multiple asynchronous boundaries before accessing databases, caches, or downstream services. If the trace ID gets lost across an async/await boundary, the trace breaks into pieces, leaving you with nothing but isolated spans and scattered logs.

Second, runtime health is increasingly impacting business experience.

A slow API isn't always caused by slow SQL. It could be due to the event loop being blocked by synchronous tasks for 200 ms, a continuous spike in V8 heap memory, garbage collection (GC) jitter, abnormal CPU usage, or process resource exhaustion. Traditional API logs struggle to answer whether the Node.js runtime itself is healthy.

Third, AI applications bring new observability targets.

More and more Node.js services are beginning to host AI capabilities. A single request isn't just HTTP + DB anymore; it might involve OpenAI calls, LangChain/LangGraph orchestration, streaming generation via the Vercel AI SDK, tool calls, embeddings, and RAG retrievals. Without AI-native observability, developers struggle to figure out if the bottleneck is in the model, the tools, the retrieval, or their own business logic.

Fourth, combining multiple tools introduces new costs and complexities.

Traditional APMs excel at APIs and databases but often overlook AI calls. AI observability tools are great at prompts, tokens, and model traces, but typically lack runtime metrics and core APM features. Meanwhile, a self-managed OpenTelemetry setup requires you to maintain exporters, plugins, sampling strategies, resource attributes, and console capabilities. As you stitch more tools together, troubleshooting paths and operational costs multiply.

This is exactly where the Node.js agent provides value. It isn't just another monitoring tool offering raw data; it is a one-time integration that brings traditional APM, AI observability, runtime health, and production configuration operations into a single troubleshooting loop.

The Solution: One Integration for Full-Stack Observability

Our solution is the Alibaba Cloud ARMS Node.js agent. Built on the core OpenTelemetry data model, it is seamlessly integrated end-to-end with ARMS. Its core design philosophy can be summarized in one sentence:

One integration, automatic instrumentation before the business code runs, effortlessly connecting your Node.js application's traces, metrics, logs, and context propagation.

The integration package is named @loongsuite/cms_node_sdk (where "cms" is a legacy naming convention), but on the product side, it functions as the ARMS Node.js agent package.

For CommonJS projects, you can use the preloading method:

ARMS_APP_NAME=your-app \
ARMS_REGION_ID=cn-hangzhou \
ARMS_LICENSE_KEY=your-license-key \
node -r @loongsuite/cms_node_sdk/register app.js

For ESM projects, you can use the Loader method:

node --experimental-loader=@loongsuite/cms_node_sdk/import-hooks app.mjs

If you prefer explicit lifecycle management within your code, you can also use the programmatic approach:

const { NodeSDK } = require('@loongsuite/cms_node_sdk');

const sdk = new NodeSDK({
  serviceName: 'your-app',
  licenseKey: 'your-license-key',
  regionId: 'cn-hangzhou',
  workspace: 'your-workspace',
});
sdk.start();

Additional configurations, such as sampling strategies, plugin toggles, and resource attributes, can be added on demand during programmatic initialization.

During startup, the agent performs several key initialization tasks: creating the Context Manager, Tracer, Propagator, Exporter, Meter, and Log Manager, and registering built-in auto-instrumentation plugins. Thereafter, as requests enter the application, database queries are executed, downstream services are called, and runtime metrics are triggered, all this telemetry is mapped to a unified observability data model and reported to ARMS, while generated logs are seamlessly injected with trace contexts for end-to-end correlation.

Six Core Capabilities to Complete the Troubleshooting Loop

Capability 1: Zero-Code Preloading for Ultra-Low Integration Costs

For many production systems, the hardest part of integrating monitoring isn't "writing a few lines of code," but rather ensuring it doesn't alter business logic, impact the startup method, or disrupt the existing engineering structure.

The ARMS Node.js agent supports two mainstream integration paths:

Project Type	Recommended Method	Characteristics
CommonJS Projects	`node -r @loongsuite/cms_node_sdk/register app.js`	Automatically loads the agent before business code execution.
ESM Projects	`--experimental-loader=@loongsuite/cms_node_sdk/import-hooks`	Injects automatically during the module loading phase.
Projects Requiring Fine-Grained Control	`new NodeSDK(...).start()`	Allows customization of sampling, exporters, plugins, and resource attributes.

This means whether you're running a traditional Express/Koa service, a BFF, a gateway, or an ESM project, you can choose the integration method that fits best.

The ESM mode uses import-in-the-middle to achieve module interception and supports automatic instrumentation for ESM dependencies. If your project contains a complex combination of loaders, we recommend verifying the module loading sequence in a testing environment first.

A special note: If you opt for the programmatic approach, the agent must be initialized before any business modules are imported or required. This ensures that HTTP, database, cache, and other modules are properly instrumented during the loading phase.

Capability 2: Automatic Instrumentation for Mainstream Frameworks and Middleware to Unify Traces

The call chain of a Node.js application is rarely a single HTTP request; it's a web composed of frameworks, middleware, databases, caches, RPCs, and message queues.

The ARMS Node.js agent comes with built-in automatic instrumentation covering core server-side paths:

Category	Supported Targets
Web and Network	HTTP/HTTPS, Express, Koa, Undici, Net, DNS
RPC and Real-Time Communication	gRPC, Socket.IO
Database	MySQL, MySQL2, PostgreSQL, MongoDB, Mongoose
Cache	Redis, ioredis
Message Queue	Kafka

When a request enters the application, the agent automatically creates a server-side span. As the request proceeds to access a database, a cache, or a downstream HTTP service, these child calls are merged into the same trace. There is no need to manually add instrumentation points throughout your business code, let alone rush to patch them after an incident occurs.

For logging scenarios, the agent injects the trace context into logging outputs like Console, Pino, Winston, and Bunyan, allowing logs and traces to be queried together within the same context.

On the ARMS console, you can view the complete path of a request from entry to downstream dependencies. Which API was slow? Which SQL query dragged? Which Redis operation was too frequent? Which downstream service timed out? Everything appears in a single, unified context.

Capability 3: Asynchronous Context Propagation to Keep Traces Truly Connected

Node.js's asynchronous model is a boon for service performance, but it's a major hurdle for distributed tracing.

The ARMS Node.js agent uses AsyncLocalStorage by default to manage context. In preload mode, it automatically downgrades to the AsyncHooks solution for older runtimes that do not support AsyncLocalStorage. It preserves the current span across asynchronous boundaries, ensuring that sub-operations within Promises, async/await, callbacks, and timers can still find their parent traces.

Additionally, the agent features built-in support for W3C Trace Context and Baggage propagation. Entry requests can extract upstream trace contexts, and exit requests can automatically inject trace headers. Thus, your Node.js service is no longer an isolated island; it can be seamlessly integrated with Java, Go, Python, frontend applications, gateways, and downstream services to form a complete topology.

When a user reports "occasional slowness in the payment API," troubleshooting no longer stops within the Node.js process. You can trace it all the way down to the database, cache, third-party APIs, and even backend microservices.

Capability 4: Runtime Metrics for Insights into the Event Loop, V8, and Process Health

Many Node.js performance issues do not immediately manifest as business errors.

If the event loop is blocked by CPU-intensive tasks, all APIs will slow down globally. A continuous increase in V8 heap memory might eventually trigger frequent GC. Abnormalities in process CPU, RSS memory, or thread pool resources could make the service unstable during peak hours.

The ARMS Node.js agent includes built-in capabilities to collect runtime metrics, covering:

Event loop delay distribution (including min, max, mean, stddev, P50/P90/P99, etc.) and utilization.
V8 heap memory and GC metrics.
Process CPU time, CPU utilization, physical memory, virtual memory, and thread count.

These metrics are periodically collected by the MeterManager and reported to ARMS via gzip + protobuf. This allows you to identify "which trace is slow" from an API perspective and "why the entire service is slow" from a runtime perspective.

The thread count provided is an estimate based on CPU cores and the libuv thread pool size, which is useful for trend observation. If exact thread counts are required, they can be supplemented via system-level or native capabilities.

This is especially critical for high-concurrency APIs, long-lifecycle services, real-time communication systems, and AI inference orchestration. Often, the true root cause isn't found in a specific line of business code, but rather in the shifting trends of runtime resource states.

Capability 5: AI-Native Observability to Expose Models, Tokens, Streaming Responses, and Tool Calls

Node.js is rapidly becoming a vital server-side runtime for AI applications. Growing numbers of teams are using frameworks and SDKs like OpenAI SDK, LangChain.js, LangGraph, Vercel AI SDK, and Anthropic Claude SDK to build intelligent customer service systems, coding assistants, data analysis agents, and internal productivity tools.

Troubleshooting AI applications differs greatly from traditional web services. You need to know:

How long does a model invocation take?
What are the input and output token counts?
Is the TTFT abnormally high?
Where did the streaming response break?
Are tool calls, RAG retrievals, embeddings, or reranks dragging down the overall trace?
In a single Agent invocation, what is the relationship between the model, tools, database, and external APIs?

The ARMS Node.js agent includes built-in AI-oriented automatic instrumentation that covers scenarios like OpenAI, LangChain, LangGraph, Vercel AI SDK, and Anthropic Claude SDK. By incorporating GenAI semantics, it captures model invocations, token usage, streaming responses, tool calls, and error details.

This means you no longer have to cross-reference model platform logs, business logs, and trace logs separately when troubleshooting your AI application. A single user query can be analyzed within the same trace, seamlessly following the flow from the Node.js API entry point, through Agent orchestration and model invocations, all the way to tool and database accesses.

Capability 6: Remote Dynamic Configuration Enables Production Troubleshooting Without Restarts

Monitoring configurations in production environments need to be agile and dynamic.

The ARMS Node.js agent supports remote configurations pushed from the console. Approximately 60 seconds after startup, the agent pulls the remote configuration for the first time, and polls it every 60 seconds thereafter. Configuration changes take effect without requiring an application restart. Currently supported dynamic capabilities include:

Adjusting sampling strategies: Full sampling, no sampling, or fixed-ratio sampling.
Adjusting span attribute limits: Controlling the maximum length and count of attributes, events, and links.
Enabling or disabling plugins: Turning off specific database, cache, HTTP, or AI plugins on demand.

This is incredibly useful for troubleshooting in production.

During traffic spikes, you can temporarily lower the sampling rate to manage costs and overhead. If a specific plugin has a compatibility risk with a certain business library version, you can disable it temporarily. If you need to debug a complex issue, you can briefly increase the sampling rate and revert it once the issue is resolved.

Monitoring systems should never be a bottleneck for business rollouts. Dynamic configuration transforms the agent from a static SDK into an operable production tool.

How Does It Compare With Common Solutions?

Comparison 1: Vs. Troubleshooting Solely With Logs

Logs are important, but logs are not traces.

Dimension	Logs Only	ARMS Node.js Agent
Request Path	Requires manual stitching	Automatically generates complete traces
Asynchronous Context	Breaks easily	Transmitted via AsyncLocalStorage/AsyncHooks
Databases and Caches	Relies on manual logging	Automatically captures critical calls
Runtime Health	Usually missing	Event Loop, V8, and Process metrics
AI Invocations	Requires custom business logs	Automatically observes models, tokens, and tool calls
Production Configuration	Requires code or environment changes followed by a restart	Dynamic push from the console

Logs are great for recording business events, whereas the agent is ideal for reconstructing system behaviors. By combining the two, troubleshooting efficiency improves significantly.

Comparison 2: Vs. Self-Assembled OpenTelemetry JS

OpenTelemetry JS is an excellent open-source standard with an open ecosystem and universal protocols. However, for enterprise users trying to implement it, there is often a whole new set of engineering challenges to resolve: Which exporter to use? How to configure sampling? Which plugins to select? How to standardize resource attributes? How to correlate logs? How to observe AI applications? And how to push dynamic configurations from a console?

The ARMS Node.js agent is built on the core OpenTelemetry data model and features end-to-end integration tailored for Alibaba Cloud ARMS. For teams already using the Alibaba Cloud observability ecosystem, it functions more like an "out-of-the-box agent" rather than a bundle of low-level components requiring manual assembly.

In short, OpenTelemetry provides standard building blocks; the ARMS Node.js agent delivers a complete, production-ready integration path.

Comparison 3: Vs. Traditional APM Node Agents

Traditional APM agents typically excel at web, database, and basic tracing. However, facing the evolving landscape of modern Node.js applications, they struggle to cover new scenarios: ESM, AI SDKs, Agent frameworks, token statistics, streaming responses, remote dynamic configuration, and cross-language semantic consistency.

The advantages of the ARMS Node.js agent include:

Deep integration with the ARMS console.
Default coverage for mainstream Node.js server-side libraries.
Support for the three core pillars of observability: traces, metrics, and logs.
Support for AI scenarios like OpenAI, LangChain, LangGraph, and Vercel AI SDK.
Remote, dynamic configurations for production environment operations.
OTLP/ARMS trace reporting, combining standardization with advanced platform capabilities.

Quick Integration: Simpler Than You Think

Prerequisites

Node.js 16.x or above is recommended. Node.js 18 or 20 LTS is recommended for production environments.

Projects using npm, yarn, or pnpm.

The build environment can access the Internet or the Alibaba Cloud intranet, and security groups allow outbound traffic on ports 80 and 443.

You have obtained your ARMS LicenseKey and Region ID.

Step 1: Install the Agent

npm install @loongsuite/cms_node_sdk

You can also use yarn or pnpm:

yarn add @loongsuite/cms_node_sdk
pnpm add @loongsuite/cms_node_sdk

Step 2: Configure Environment Variables

export ARMS_APP_NAME=your-app
export ARMS_REGION_ID=cn-hangzhou
export ARMS_LICENSE_KEY=your-license-key

The agent is also backwards compatible with legacy environment variables using the CMS_ prefix, making it easy for existing teams to migrate gradually. For new projects, we recommend using the ARMS_ prefix exclusively.

For Docker environments, add these to your Dockerfile:

ENV ARMS_APP_NAME=your-app
ENV ARMS_REGION_ID=cn-hangzhou
ENV ARMS_LICENSE_KEY=your-license-key

Step 3: Start the Application

For CommonJS projects, preloading is recommended:

node -r @loongsuite/cms_node_sdk/register app.js

For ESM projects, using a Loader is recommended:

node --experimental-loader=@loongsuite/cms_node_sdk/import-hooks app.mjs

When programmatic control is needed:

const { NodeSDK } = require('@loongsuite/cms_node_sdk');

const sdk = new NodeSDK({
  serviceName: 'your-app',
  licenseKey: 'your-license-key',
  regionId: 'cn-hangzhou',
  workspace: 'your-workspace',
});
sdk.start();

Step 4: Verify Data

After the application starts, you'll be able to see the integrated application within about one minute on the ARMS console under "Application Monitoring > Applications". By entering the Application Details page, you can view the application topology, API invocations, trace links, SQL analysis, runtime metrics, and more.

Performance and Overhead: Monitoring Should Help Your Business, Not Slow It Down

The true value of a monitoring SDK is to help discover problems, not to become a problem itself.

By design, the ARMS Node.js agent follows the principles of being "low-intrusion, sample-enabled, switchable, and recoverable":

Mechanism	Purpose
Batch Export	To reduce network requests and export frequency
gzip + protobuf	To minimize data transmission size
Sampling Strategies	To control trace data volume in high-traffic scenarios
Plugin Toggles	To enable only the specific collection capabilities required by the business
Exception Protection	To ensure automatic instrumentation failures do not affect the main business flow
Shutdown/Unpatch	To gracefully close and revert patches when the application exits
Remote Configuration	To allow dynamic parameter tuning in production without restarts

Furthermore, the agent enables environment, process, host, and Kubernetes resource detection by default. This automatically populates resource attributes like service name, host, process, container, and workload, eliminating the manual overhead of maintaining labels post-integration.

When the application exits, the preload mode listens for SIGINT and SIGTERM signals. It then calls shutdown() to sequentially close the automatic instrumentation plugins, TracerManager, MeterManager, and LogManager, ensuring buffered data is flushed and patches are safely reverted.

In standard business scenarios, the agent's impact on application performance is minimal. For highly concurrent or highly sensitive traces, we recommend combining load-testing results to set a reasonable sampling rate and turn off unused plugins as needed.

Applicable Scenarios

Enterprise-Grade Node.js Web Services

Ideal for Express, Koa, BFF, API gateways, and internal corporate systems, helping teams rapidly build API performance, error rate, dependency call, and topology views.

Microservices and Distributed Systems

Perfect for systems with numerous services, complex downstream dependencies, and requirements for cross-language tracing. Via Trace Context and Baggage propagation, your Node.js services can join Java, Go, Python, and other services to form a complete trace network.

Database and Cache-Intensive Applications

Designed for systems heavily utilizing MySQL, PostgreSQL, MongoDB, Redis, and ioredis. Slow SQL queries, cache hotspots, and slow downstream dependencies are all merged into a single request trace.

AI/Agent Server-Side Applications

Suitable for intelligent customer service, AI coding assistants, RAG Q&A, and data analysis Agents. It tracks OpenAI, LangChain, LangGraph, and Vercel AI SDK invocations to analyze tokens, tool calls, streaming responses, and model execution times.

Long-Lifecycle Node.js Services

Ideal for real-time communication, queue consumption, background tasks, and daemon workers. Runtime metrics assist in diagnosing blocked event loops, memory bloat, GC anomalies, and process resource exhaustion.

Production Systems Requiring Dynamic Operations Monitoring

Made for mission-critical businesses where frequent restarts are out of the question. Sampling, span limits, and plugin toggles can be dynamically pushed from the console, allowing monitoring strategies to adapt instantly to current business states.

Conclusion

Node.js has made server-side development incredibly efficient and flexible, but it has also ushered production troubleshooting into a far more complex era. Asynchronous contexts, massive ecosystem modules, database and cache dependencies, runtime health, and AI traces—any layer could hide the root cause.

The goal of the ARMS Node.js agent is simple: To make Node.js observability as straightforward as integrating an npm package.

A single integration automatically covers HTTP, frameworks, databases, caches, RPCs, message queues, logs, runtimes, and AI calls. A single trace connects user requests, service logic, and downstream dependencies. A single console dynamically manages sampling, plugins, and data reporting strategies.

Server-side distributed tracing is now fully within your reach.

Try It Now: Log on to the Alibaba Cloud ARMS console, create an application monitoring integration configuration, obtain your LicenseKey, and start integrating the Node.js agent today.

Technical Support: If you encounter any issues during the integration process, feel free to connect with the Alibaba Cloud Observability Team via our DingTalk support group.

Seeing Every User Step: How Session Replay and Heatmaps Drive Evidence-Based UX Optimization

ObservabilityGuy — Mon, 24 Aug 2026 05:37:26 +0000

This article introduces Cloud Monitor's Session Replay and Heatmap capabilities for visually tracking user behavior to troubleshoot frontend issues and optimize UX.

Introduction

As frontend user experience (UX) optimization becomes increasingly sophisticated, the challenge for developers has evolved from merely "detecting errors" to truly "understanding user behavior." Traditional, metrics-only monitoring often falls short when tackling "black-box" challenges—such as UI lag without diagnostic logs, or dropping conversion rates with no obvious cause.

As a unified observability platform, Alibaba Cloud's CloudMonitor Service (CMS) 2.0 has been continuously deepening its capabilities in Real User Monitoring (RUM). To help developers pierce through client-side blind spots, the CMS team has introduced Session Replay and 3D Heatmap capabilities.

Powered by incremental DOM recording and multidimensional behavioral analysis, CMS captures and reconstructs the exact visual context of user interactions. Combined with a robust four-tier privacy protection mechanism, it enables developers to seamlessly transition from reproducing individual edge cases to gaining aggregate behavioral insights in a fully compliant manner—completing the UX optimization loop by truly "seeing every user step."

A user reports, "The page seems to have frozen," but you comb through the logs and find no errors. A product manager worries about the conversion funnel but has no idea which button made users hesitate. A customer service ticket reads, "I can't click anything," yet the page works perfectly when you open it. These frontend experience mysteries unfold in countless teams every day. Session Replay and Heatmaps are the ultimate weapons to end this guesswork—one lets you return to the "scene of the incident," and the other reveals "group behavior patterns."

Three Blind Spots in Frontend Experience

Backend observability already boasts mature tracing, logging, and metrics monitoring systems. However, when issues occur in the browser—the place closest to the user but farthest from the developer—we often hit a blind spot.

Blind Spot 1: What Exactly Did the User Experience?

A user writes in a ticket, "I clicked the order button, but nothing happened." You check the code, and the logic is fine; the API logs show no invocation records, and the monitoring dashboard looks normal. Was the button obstructed? Did a JavaScript error occur? Was there a network timeout? Or did the user simply miss the click? You cannot reproduce the issue because you never "saw" the page exactly as the user did.

Blind Spot 2: Where Did the User Hesitate?

The conversion rate dropped by 3% after a product revamp, but A/B testing only tells you that it is "worse," not "where it is worse." Are users struggling to understand the new navigation layout? Is the price tag not prominent enough? Or is the CTA button placed unintuitively? Without behavioral data, optimization strategies are left to guesswork.

Blind Spot 3: How Did the Page Actually Perform?

Performance monitoring tells you the LCP is 2.3 seconds, but the "slowness" perceived by the user could stem from a blank first screen, flickering during image loads, or janky scrolling. A single metric cannot capture the complete picture of the user experience.

Session Replay and Heatmaps are the two purpose-built tools designed specifically to eliminate these three blind spots. The former recreates the scene of each individual incident, while the latter reveals group patterns. Together, they deliver the full capability to "see every user step."

1. Session Replay: Recreating the Scene of the Incident

The core idea of Session Replay is simple: since you cannot ask the user to reproduce the issue for you, "record" the user's operation process so developers can review it themselves.

Recording here does not mean actual screen recording. The browser SDK reconstructs the user's operation process based on DOM snapshot capture and incremental mutation tracking mechanisms. It records a series of structured DOM event sequences rather than massive video files. This approach not only reduces data volume by an order of magnitude but also achieves pixel-perfect reproduction of page structures and interaction details during playback. It also supports advanced debugging features like arbitrary timeline jumps and local magnification, greatly improving troubleshooting efficiency.

Standard Session Replay view: The left panel lists user events like click and navigation chronologically, accurate to the timestamp. The central area provides a 1:1 reproduction of the page exactly as the user saw it. The bottom timeline supports video-like fast-forward and rewind. The right pane can pinpoint DOM structure changes at any moment. A single replay reveals the full picture of the issue.

What Does It Record?

Simply put, everything the user sees and does on the page:

DOM mutations: Additions, deletions, and modifications to the page structure, style changes, and dynamic content loading.
User interactions: Clicks, scrolling, typing, and form operations.
Page status: Focus/Blur (whether the tab is in the frontend) and visibility change.
Route changes: Complete page changes during SPA route switches, supporting both History and Hash modes.

How Is Data Transmitted? Segmented Upload + Three-Tier Compression Fallback

Uploading recorded data in real time would place unnecessary stress on the network and battery. Real User Monitoring (RUM) uses segmented upload: The SDK first buffers data locally in segments and performs a flush every 200 events or every 5 seconds (whichever comes first). When the page becomes hidden, frozen, or unloads, the SDK immediately flushes to ensure no data is lost. A single session records for a maximum of 1 hour before automatically segmenting.

The pre-upload compression process is a prime example of the engineering effort behind this system. We adopted a three-tier fallback strategy to strike a balance between performance, compatibility, and reliability:

Priority	Solution	Applicability	Advantage
① First choice	Browser native `CompressionStream` API	Chrome 80+ / Safari 16.4+ / Firefox 113+	Zero additional overhead, built into the browser
② Fallback	Web Worker + pako deflate (inlined at build time)	All mainstream browsers	Does not block the main thread; falls back again after a 30-second timeout
③ Last resort	Raw data upload	Constrained environments	Ensures data integrity, even at the cost of speed

The essence of this mechanism is that a failure at any tier gracefully falls back to the next—enjoying the high efficiency of new APIs without abandoning any legacy users.

What about performance impact? Empirical tests show that Session Replay incurs a CPU overhead of 1–3% and a memory overhead of 2–5MB on standard pages (depending on page complexity). The sampling rate configuration allows you to precisely control the recording scope—10–20% is recommended for production environments, while you can set it to 100% for testing environments.

Privacy Protection: Four-Tier Security Strategy

Recording user operations inevitably involves privacy. The RUM SDK provides a four-tier privacy protection configuration, ranging from the strictest to the most lenient:

Privacy Level	Description	Applicable Scenarios
`mask` (Default)	Masks all text content and input boxes	General scenarios, the most secure choice
`user-input`	Masks only user input content (passwords, input boxes, etc.)	Scenarios where page text information needs to be retained
`allowlisted`	Records only elements specified in the allowlist, and masks other content	Fine-grained control where only specific areas need to be recorded
`allow`	No masking is performed; the complete page is recorded	Internal systems, pages without sensitive information

In addition, it supports granular control via CSS class names: elements marked with .rum-block are completely masked (black blocks), elements marked with .rum-ignore are not recorded, and text marked with .rum-mask is masked. Compliance and observability can go hand in hand.

When Is It Most Useful?

Scenario 1: Bug Reproduction. A user reports an intermittent UI anomaly. Previously, you would have to ask the user to "try again and send me a screen recording." Now, you can directly find the corresponding Session Replay in the backend and fast-forward to the exact moment of their operation—DOM structures, style changes, and interaction sequences are crystal clear.

Scenario 2: Conversion Funnel Analysis. An e-commerce checkout flow has 5 steps, and 40% of users drop off at step 3. By replaying the Sessions of these churned users, you discover that a mandatory field in the address form at step 3 is obscured by the keyboard on small-screen devices—an insight impossible to glean from logs and metrics alone.

Scenario 3: Customer Support. A user calls to say, "I spent ages filling out the form, but I can't submit it." Customer service uses the Session ID to find the replay and observes the user repeatedly clicking the date picker to no avail—it turns out the date format prompt was not obvious enough. Pinpoint the issue in 30 seconds, eliminating back-and-forth communication.

2. Heatmaps: Bringing Group Behavior to the Surface

While Session Replay helps you see "the story of one person," a Heatmap helps you see "the pattern of a group." When hundreds or thousands of users leave behavioral traces on the same page, the Heatmap aggregates and overlays this data, intuitively displaying the focus of user attention and the distribution of behavior using color intensity.

RUM offers three dimensions: Click Heatmap, Area Heatmap, and Scroll Heatmap. These answer questions from different angles: "Where are users clicking? What are they paying attention to? How much are they seeing?"

2.1 Click Heatmap: The User's "Mouse Fingerprint"

The most classic and intuitive Heatmap type. During the capture phase, the SDK listens for page click events, collecting the document coordinates (x, y), relative coordinates within the element (ex, ey), selector path, and viewport size for each click. After the SDK reports the data, the backend overlays all click coordinates onto the page snapshot, using color intensity to indicate click density—red areas are where users love to click the most, while blue areas are neglected "cold zones."

A Click Heatmap of a real e-commerce homepage: The "Interaction Statistics" in the upper right corner shows 1,817 total clicks and 987 page views—averaging nearly 2 clicks per visitor. The "Element Clicks" on the left automatically categorizes elements: input 244 times (13.43%), img 156 times (8.59%), and button 144 times (7.93%), telling you which element types users touch most frequently. The blue-to-red color blocks and numeric rankings on the central page clearly mark the popularity of each clicked area.

RUM's click collection goes beyond "recording coordinates" and performs two tasks rarely seen elsewhere:

Determining element interactivity (target.reaction): Tags such as button, a, input, select, option, textarea, details, summary, audio, and video, or elements with onclick/role=button/tabindex/cursor:pointer attributes, are marked as "interactive." If a non-interactive element receives a large number of clicks, it means users think it should be clickable—this highlights a UI design issue.
Determining click trustworthiness (target.trust): Multi-dimensional evaluation based on event source (isTrusted), element size (less than 20px is considered untrustworthy), and visibility (display:none/visibility:hidden/opacity:0). Filtering out automated scripts and accidental touch noise ensures the Heatmap genuinely reflects user behavior.

Invalid Click Analysis—An underestimated capability. When numerous clicks occur where reaction=0 (non-interactive elements), it means users are clicking on elements that don't respond—this typically exposes UI design issues: elements that look like buttons but aren't, or images that appear clickable but aren't. Through Invalid Click Analysis, product teams can precisely identify and fix these "interaction illusions," turning invisible friction into a smoother, more intuitive experience.

2.2 Area Heatmap: The User's "Attention Map"

If the Click Heatmap answers "where users clicked," the Area Heatmap answers "what users are paying attention to." Building on click data, it incorporates element sizes and positions to map user attention to various business areas on the page.

The essence of the Area Heatmap lies in its business readability: blue dashed boxes outline key business modules like product cards, navigation menus, and campaign entries, with each area displaying click counts, percentages, and business user numbers. Operating teams no longer need to count pixels; they can directly see "which Banner drove 30% of clicks" or "which product card was largely ignored"—shifting the basis for decisions from "feeling" to "numbers."

For large banners, image walls, or content lists, the Area Heatmap tells you which areas "catch the eye" and which are "virtually useless." It serves as a "yardstick" for A/B testing before and after revamps and for evaluating the value of homepage placement slots.

2.3 Scroll Heatmap: How Much Did Users Actually See?

This is a deceptively simple but critical question—and the Scroll Heatmap is what finally gives you the answer.

Scroll depth distribution at a glance: 100% of users (31 people) saw the first screen, but by the 54.8% scroll depth mark, only 17 people were still scrolling—meaning nearly half of the visitors never made it past the midpoint of the page. The color band on the right, ranging from red (high reach rate) to green (low reach rate), visualizes the "drop-off point" for each percentile. If your core CTA button unfortunately lands in the green zone, it is virtually hidden from half of your users.

Instead of treating scrolling as discrete Action events (which would generate a massive volume of events), the RUM SDK continuously tracks scroll depth throughout the lifecycle of each View, reporting the final statistical summary when the View ends:

Field	Meaning
`scroll.max_depth`	Maximum scroll depth reached by the user (percentage)
`scroll.max_depth_scroll_top`	Scroll position when the maximum depth is reached (pixels)
`scroll.max_scroll_height`	Maximum scroll height of the page
`scroll.max_scroll_height_time`	Time when the maximum height is reached

Core calculation formula: scrollDepth = (scrollTop + clientHeight) / scrollHeight.

To prevent performance issues caused by high-frequency scroll events, the SDK throttles using throttle (leading + trailing mode), sampling at most once every 100ms. Concurrently, it uses ResizeObserver to listen for document.body height changes, covering dynamic scenarios like lazy-loaded images, infinite list prefetching, and accordion panel expansions to prevent underestimating max_scroll_height. When the View ends, the SDK actively cancels tail calls to eliminate cross-View data leakage.

For content operation teams, this Heatmap is a "content visibility map"—if you place your most important CTA button outside the red zone, it is almost equivalent to hiding it.

3. Differentiation from Competitors

Session Replay and Heatmaps are not new concepts; vendors like Datadog and Sentry both offer them. However, their focuses and maturity levels differ significantly. The following table compares these two core capabilities:

Capability Dimension	Datadog	Sentry	Alibaba Cloud RUM
Session Replay	✅	✅	✅
Click Heatmap	✅	❌ Explicitly not supported	✅
Area Heatmap	❌	❌	✅
Scroll Heatmap	✅	❌	✅
Privacy Protection	Data masking	Text masking	Four-level configuration
Data Compression	Not detailed	Not detailed	Three-tier fallback strategy (Native -> Worker -> Last resort)
SPA Support	✅	✅	✅ History + Hash
Invalid Click Detection	✅ Rage click	❌	✅ Reaction + Trust
Web Vitals	✅	✅	✅ LCP/CLS/FID/INP

Our Differentiating Advantages

First, comprehensive coverage of three Heatmap types. Very few vendors simultaneously offer Click, Area, and Scroll page-level Heatmaps. RUM's three-in-one toolkit paints a full picture of user behavior across three dimensions: "click locations," "attention areas," and "browsing depth."

Second, granular click quality analysis. Through cross-analysis of reaction (interactivity) and trust (trustworthiness) dimensions, RUM not only tells you "where users clicked," but also "whether these clicks were valid" and "whether they came from real users."

Third, four-tier privacy protection configuration. Ranging from default mask, through granular allowlisted, to fully open allow, this covers the complete spectrum from the strictest compliance needs to the most permissive debugging needs.

Fourth, a three-tier compression fallback strategy that balances compatibility, performance, and reliability. It prioritizes the native CompressionStream API, falls back to a Web Worker running pako, and guarantees delivery via raw data as a last resort. This meticulous engineering consideration delivers clear advantages in large-scale production deployments—fast execution on new browsers, reliable execution on older browsers, and zero data loss in extreme conditions.

4. Get Started in Five Minutes

Is integration complicated? Not at all. If you are already using the Cloud Monitor Browser SDK, you only need to add a few lines to your initialization configuration:

Simultaneously Enabling Session Replay + Heatmap

import armsRum from '@arms/rum-browser';
armsRum.init({
  endpoint: 'https://your-endpoint.com/rum/web/v2',
  // Session Replay configuration
  replay: {
    enable: true,
    sampling: 20,            // 20% of sessions will be recorded
    privacy: {
      level: 'user-input',   // Mask only user inputs
    },
  },
  // Heatmap configuration
  collectors: {
    click: {
      enable: true,
      trackUserInteractions: true,  // Enable Heatmap data collection
    },
  },
});

It is that simple. The Replay sampling rate takes effect at the session level—once a session is selected for recording, it will continue recording throughout the entire session. Heatmap collection is controlled independently; you can enable 100% click collection to obtain complete Heatmap data.

Recommendations for sampling rates: We recommend setting the Replay sampling rate to 10–20% in production environments to control storage costs while ensuring adequate issue coverage. For Heatmaps, we recommend keeping trackUserInteractions at 100% sampling, because Heatmap's value comes from aggregate analysis—the more data you have, the more accurate the results. These two sampling rates are configured independently and don't need to be kept in sync.

5. From "Guessing" to "Seeing"

The core challenge of frontend experience optimization has never been "not knowing what to optimize", but rather "not knowing where to start".

Session Replay allows you to navigate the page exactly as the user did, precisely pinpointing that frustrating moment.
Click and Area Heatmaps provide a high-level overview, clearly revealing the behavioral patterns of user groups.
Scroll Heatmap allows you to answer the simplest yet most critical question: "How many people actually saw the content we carefully crafted?"

This combination covers the complete process from individual case diagnosis to group insights. You no longer need to ask users to "try again and send me a screenshot", and you no longer need to argue endlessly in meetings over whether "users actually like this redesign".

When your observability system covers the complete process from backend to frontend, and from metrics to behaviors, those experience issues once hidden in user browsers will have nowhere to hide.

Experience Now: Go to Cloud Monitor to create a RUM application, obtain the endpoint, and start integrating in minutes.
Technical Documentation: Browse the complete configuration guide.
Community: Join our DingTalk group to connect with the Alibaba Cloud Observability Team.

STAROps RUM Inspection in Practice: Spotting Experience Degradation Early

ObservabilityGuy — Fri, 21 Aug 2026 09:57:06 +0000

This article introduces STAROps RUM Inspection, an AI-driven capability that proactively detects and analyzes subtle user experience degradations before traditional alerts fire.

Online stability has its blind spots.

Half an hour after a release, no alerts have fired, and the error rate hasn't crossed the threshold. Still, the checkout conversion rate has dipped slightly, the mobile page is taking a bit longer to render on first load, rage clicks on buttons have ticked up, and the P95 latency on one API has crept up just a bit. Looked at individually, each metric seems like a small enough blip to shrug off for now.

The problem is, users don't experience a product as a set of metrics. What they encounter is a page that won't load, a button that goes silent when clicked, a submission that seems to hang forever. Some bounce. Some retry. Some end up filing a complaint with customer support.

This is exactly the gray zone Real User Monitoring (RUM) Inspection is built to handle. Running on a fixed cadence, it cross-analyzes page performance, API latency, user behavior, crashes, conversions, and version changes against the same target entity — helping you tell, as early as possible, whether the user experience is genuinely degrading or just hitting a temporary hiccup.

What to Look for in RUM Data

RUM looks at what users actually experience under real-world conditions. Devices, networks, browsers, versions, pages, how quickly a page first renders, how responsive it feels after a click, API latency, resource failures, and any stuttering throughout the entire visit — all of it leaves a trace.

Here are a few common RUM metrics:

Largest Contentful Paint (LCP) measures how long it takes for the core content to appear. When this metric worsens, users immediately perceive the page as slow.
Interaction to Next Paint (INP) measures interaction responsiveness. If a button doesn't respond to a click, or the page freezes after input, this is the metric to check.
API P95 reveals your worst-case request latency. Even if the average looks fine, the slowest 5% of your traffic may already be struggling.
Slow sessions reflect whether the entire visit felt smooth. A single slow moment might be tolerable, but when the whole flow drags, completion rates take a hit.
Session Replay, heatmaps, and rage clicks give you visual proof, showing you exactly where users are getting stuck.
Crashes and exceptions signal that the user flow has already been broken. These need to be analyzed alongside version, device, page, and symbolication data.

While rich monitoring metrics preserve the scene of the incident, they do not automatically translate into a diagnosis. The more data you have, the heavier the subsequent workload becomes: you must continuously inspect everything, confirm whether changes point to a single issue or isolated fluctuations, and piece together metrics, samples, and behavioral evidence into actionable clues.

STAROps is a cross-domain AIOps platform from Alibaba Cloud, powered by LLMs and AI Agent technologies. It combines cross-domain observability data with LLM reasoning capabilities, overcoming the limitations of traditional Ops tools, such as steep learning curves and stubborn data silos. It lets users define objectives in natural language, while AI Agents autonomously handle the entire closed loop of dynamic planning, secure execution, and result validation. Using long-running tasks (Missions), digital employees (Agents) can automatically carry out operations like inspections, changes, and analyses—either on a schedule or triggered by events—escalating to Human-in-the-Loop (HIL) when needed.

RUM inspections, combined with STAROps missions, deliver two major capabilities: automated analysis triggered by alerts, and periodic reports informed by root cause analysis (RCA)—including inspection reports, issue summaries, and alert analysis summaries. This directly addresses the challenges outlined above.

The Boundaries Between Alerts, Inspections, and Dashboards

Alerts are ideal for catching deterministic failures. If an API goes down, the error rate clearly crosses the threshold, or a core flow fails at scale, these events should trigger an immediate response—notification, escalation, and mitigation.

Dashboards answer the question, "What is the current status?"—covering traffic, latency, error rates, and version distribution. Users can check any metric at any time, with trends visible at a glance.

Inspections cover the gray zone further upstream in the pipeline: a single metric might not be severe enough to trigger an alert, but multiple signals are already trending downward on the same target. Compared to alerts and dashboards, inspections focus more on interpreting fused signals and compounding degradations: which signals are moving together, what target they're hitting, which users are affected, and who should take it from here.

Take /checkout as an example. After a new release goes live, it doesn't go down entirely, and the error rate doesn't spike. However, mobile users see slower LCP, worse INP, elevated P95 on payment/create, more slow sessions, increased rage clicks, and a lower conversion rate. Isolate any one of these, and you could write it off as a mere fluctuation; put them all together, and it becomes very hard to ignore.

Inspections are periodic tasks, so you don't need a lengthy report every time. Run an hourly sweep to catch which objects are starting to drift from baseline; run a daily interpretation to piece together the chain of evidence behind combined degradations and suggest next steps; and run a weekly rollup that flags issues with tail latency or frequent recurrence, adding them to the remediation backlog.

Object-Based Inspections

Traditional metric-based analysis often fragments problems. You might see a slow page here, a sluggish API there, and an error signature somewhere else. Each looks alarming in isolation, but scattered like this, they're hard to prioritize—loud, but not necessarily urgent.

Inspections need to flip this approach: pin down the object first, then examine the metrics.

An object can be a page, a business path, a version, a device type, a region, a channel, or some combination of these. Only once the object is pinned down do the metrics have something to anchor to. Otherwise, "LCP increased by 8%" is just a data point; but "/checkout + v2.8.1 + mobile LCP, INP, API p95, slow sessions, rage clicks, and conversion rate all degrading at once" is clearly worth investigating.

This step is a common source of misjudgment. A page with only a few dozen visit samples shouldn't be judged by the same standard as a core page with hundreds of thousands of daily visits. There's also the question of timing: a change at 14:00 today shouldn't just be compared to 13:00 today—you need to check it against the same time yesterday, the same time last week, and the windows before and after a release.

Dimensions must also map to specific troubleshooting actions. If mobile performance degrades, break it down by device model, OS, browser, region, and version. If the API P95 spikes, track down which slice of your slowest traffic ruined the experience. If conversions drop, dig back into how users behaved—how long they waited, where they rage-clicked, and where they dropped off.

Ultimately, the evidence has to come together. A single degrading metric only signals a fluctuation; but when business results, performance metrics, request latency, user behavior, and Session Replay all point to the same object, the conclusion holds up.

Two Types of Easily Missed Issues

The first type is when "business metrics weaken before technical metrics blow up."

For example, the payment completion rate drops by 3%, but the error rate doesn't budge, and no alerts fire. If you only watch the error count in this scenario, the problem can slip right past you. The right approach is to lay out the entire payment path in a single view: entry page load time, submit button responsiveness, payment API P95, version distribution of slow sessions, and buttons with rage clicks.

If these signals show up together, and Session Replay shows users waiting longer and longer after submitting, clicking repeatedly, or going back to retry, then a report that simply says "conversion is fluctuating" falls short. It should state plainly: the issue is concentrated in the new mobile version of the payment flow, showing up mainly as tail latency and interaction delays. R&D engineers should prioritize the tail latency on the payment creation API, while also checking button feedback and anti-duplicate submission logic.

The second type is "persistent tail latency on low-end devices."

This kind of problem tends to stay quiet. Day by day, it's barely slower; week by week, the pattern holds. On low-end Android devices, long tasks pile up, INP stays poor, the share of slow sessions runs higher, the bounce rate ticks up, and the completion rate dips slightly. It's not worth waking anyone up at 3 a.m. for, but left alone long enough, it becomes an ongoing experience tax paid by a specific group of users.

Inspections are well suited to surface exactly these kinds of issues: how broad the impact is, how long it's been going on, where it ranks in the remediation backlog, and who ultimately owns the fix.

Automatically Parsing Crashes and Aggregating High-Frequency Root Causes

However, many crash tools just spit out a raw stack trace. You can tell an error occurred, but you have no idea which version, page, or piece of code to chase down. That's why, once crashes are fed into inspections, the system must first perform normalization and aggregation: clustering similar exceptions and stack traces, then rolling them up by page, version, platform, device, browser, WebView, and release window. The report should then prioritize highlighting high-frequency root causes and the scope of impact.

Automated parsing has a prerequisite: symbol files must accompany the release. For the frontend and web, upload source maps that match the build artifacts; for Android, upload the corresponding version's mapping.txt; for Native, retain the corresponding debug symbols. If a file is missing, the report will only show minified line and column numbers, or obfuscated class names. Only when symbols are matched can errors be traced back to source code files, methods, activities, adapters, or click callbacks. RUM supports uploading files like source maps or mapping.txt via CLIs. For details, refer to Upload RUM symbol table files by using CMS2 CLI.

Crash interpretations should also cut down on the guesswork. Take Android's IndexOutOfBoundsException, for instance. The report shouldn't just say "index out of bounds"; it needs to clarify that it happened after the user clicked a list item, accessing an index that exceeds list.size(). It should also include the affected versions, device distribution, sample count, user count, and suggested troubleshooting directions. For frontend errors caused by accessing undefined properties, the report should pin them down to specific components, API fields, or canary resource versions.

The symbol files themselves also need to be managed. Since source maps might contain source code information and mapping.txt may expose code structure, they're better kept in a controlled repository, tightly keyed to application, environment, version, build number, and resource hash. This way, the moment a crash occurs, the matching symbol file can be automatically located, and the report can reliably surface high-frequency root causes instead of just leaving a pile of raw, unreadable stack traces.

Default Reports and Custom Reports

Inspection reports support multiple formats. By default, they cover four common types: hourly reports spot objects just starting to deviate from the baseline; daily diagnostic reports explain recurring degradations throughout the day; weekly reports roll up issues with tail latency that recur frequently and belong in the remediation backlog; and full RCA inspections perform a complete root cause analysis on a specific issue, stringing together the timeline, impact scope, evidence chain, root cause judgment, handling suggestions, and review criteria.

Regardless of the type, a report should consistently cover a few key components. First, provide the conclusion: which business path, version, device type, or user group was actually affected this time. Next, clearly define the impacted objects: page, API, version, platform, region, user scale, and business path—each one spelled out precisely. Then comes the combined evidence, explaining which signals degraded together and how they compare to the baseline. Granular evidence must follow—Session Replay, heatmaps, sample sessions, and error samples are there to support the conclusion, not just to pad the report with visuals. Finally, wrap up with handover suggestions: should R&D engineers check the API or the interaction first? How large an impact radius should SREs keep monitoring? Which conversion funnel should product managers track? And which metrics should they review, and how soon?

Users can also customize reports based on their own scenarios. Building on existing reports, clearly communicate a few requirements to the Agents: is the inspection target a page, an API, a version, or a business path? Is the time window hourly, daily, or pre/post-release? Do the key metrics focus on performance, exceptions, conversions, behavior, or crashes? Should the output lean toward a handover ticket, a post-mortem report, a remediation backlog, a daily risk report, or an RCA? The goal of customization isn't to write a few extra paragraphs, but to drive down the communication costs of the next steps. A truly usable report will nail down the troubleshooting targets, the basis for judgments, the owners, and the review criteria.

Quick Start

Prerequisite: Integrate RUM. Then, log on to the STAROps console. Alternatively, you can head straight to the interactive demo and click RUM inspection to try it out.

In the STAROps console, click Mission on the left, then click New mission. In the dialog box that appears, click the RUM inspection card. Wait for it to output the inspection plan, then just type your confirmation.

In the inspection dialog, you can modify the prompt at any time. Just make sure to clarify: the target objects, time windows, scenarios, and output formats.

For release inspections, you can write something more specific based on your needs:

Inspect the xxx app's /checkout payment flow, run it once an hour, and compare it with the same period before the release. Focus on mobile LCP, INP, API P95, slow sessions, rage clicks, and conversion changes. Output the risk objects, evidence chain, and handover suggestions.

For campaign monitoring or user feedback scenarios, the prompt can be tailored to the specific situation:

Over the past hour, users have reported that tapping into the campaign page gets no response. Focus on mobile, low-end Android devices, primary regions, and the new version. Combine this with rage clicks, long tasks, API latency, and Session Replay samples to output the handover suggestions and next steps for troubleshooting.

Once the task is created, first verify whether the objects and time windows in the report are correct. If the scope is too broad, narrow down the page, version, platform, or region and run it again. The more your prompt reads like a real-world problem, the easier it is for the report to lead directly into troubleshooting, and the less likely it is to churn out a bloated, exhaustive list of metrics.

How to Read the Reports

Different templates require different reading approaches, but always confirm three things first: what happened, what's the basis for this judgment, and who takes over next.

The hourly report is more like a handover ticket during an on-call shift; it answers the question, "Which objects started acting up in the last hour?" On the first screen, scan the health status, risk levels, and top risk objects, then scroll down to verify the metrics and samples supporting the conclusion.

Structurally, the hourly report hits three layers: conclusions at the top, evidence in the middle, and risks at the bottom. After reading it, the on-call engineer should at least know whether to engage, which object to target, and what to monitor for the next hour.

The daily diagnostic report rounds up weak signals that recurred throughout the day, have a more stable impact radius, and are ripe for a post-mortem. While the hourly report leans toward "should I handle this right now?", the daily report is more about "which issues from today need to be discussed?"

How Our Team Uses RUM Inspection

Our team's approach is quite simple: let alerts continue to catch issues that have clearly blown up, and use RUM Inspection for daily health checks. We run a full RCA inspection every day. When reviewing the report in the morning, we don't get lost in the weeds; instead, we first check if any new risk objects have surfaced.

In day-to-day operations, the most common scenario isn't actually "an alert fired," but rather issues that haven't yet hit the alert threshold—a page is noticeably slower than the same time last week, a specific version has a persistently high ratio of slow sessions, or a certain type of crash happens every day but in small daily volumes. Once these kinds of issues cluster on a critical path, they get pulled out for dedicated review.

When we confirm an issue needs handling, we directly create an issue ticket. We don't write generic descriptions like "page performance degraded." Instead, we clearly outline the objects and evidence: which page or flow, which versions and platforms are affected, which baseline it deviated from, what sample sessions, heatmaps, error samples, or crash aggregations support this, who should examine it first, and which metrics to check during the review.

Once a week, we review the weekly report. The weekly report doesn't rehash the daily diagnostic process; it only compares this week against last week: what new issues surfaced, which problems persisted, which ones recovered, and which high-frequency root causes kept resurfacing. By the end of the week, the team knows whether they're just chasing the same recurring bugs, or whether their remediation efforts are actually moving the needle.

Conclusion

The value of RUM Inspection doesn't lie in adding yet another set of reports. What it actually does is organize the weak signals from real user experiences into actionable insights. Alerts tell you where the line has already been crossed; inspections fill in the blind spot before that line is crossed: which objects are degrading, whether the evidence points to the same root cause, and who should take the baton next.

Once crash parsing, report generation, issue tracking, and weekly report comparisons are all linked together, experience issues are no longer scattered across dashboards, alerts, and user feedback. The team can spot degradations much earlier and confirm with certainty whether they've actually been fixed.

Try it out now: head over to Cloud Monitor 2.0 to create a RUM application—once you have your endpoint, you can start integrating RUM with your app. Visit the interactive demo to try creating a RUM inspection.

Alibaba Cloud Recognized in the Challengers Quadrant of the Gartner® Magic Quadrant™ for Observability Platforms

ObservabilityGuy — Fri, 21 Aug 2026 03:14:37 +0000

Alibaba Cloud has been recognized in the Challengers Quadrant of the Gartner® Magic Quadrant™ for Observability Platforms.

Global research and advisory firm Gartner recently published its 2026 Magic Quadrant™ for Observability Platforms report. Alibaba Cloud has been officially recognized in the Challengers Quadrant, standing out as the only vendor from the Asia-Pacific region to achieve this distinction.

As observability reaches a critical inflection point in its evolution toward Agentic Ops, Alibaba Cloud, powered by its unified observability data foundation, Cloud Monitor (CMS) 2.0, and its comprehensive AIOps platform,STAROps,empowers AI Agents to autonomously execute the entire IT operations lifecycle—from root cause analysis to closed-loop resolution—24/7. This capability underscores Alibaba Cloud's formidable competitive edge in the global AIOps market.

As enterprise-grade Agent applications are deployed at scale, observability is evolving from a passive troubleshooting tool into core infrastructure that proactively drives business resilience. Traditional siloed monitoring simply cannot support the complex data correlation required across multiple Agents, systems, and domains. Today’s enterprises need more than just basic visibility; they require a robust AIOps foundation capable of autonomous comprehension and pinpoint accuracy.

(Image: Alibaba Cloud recognized in the Challengers Quadrant of the Gartner Magic Quadrant™ for Observability Platforms)

Alibaba Cloud's CMS 2.0 consolidates previously standalone services—Cloud Monitor (CMS), Simple Log Service (SLS), and Application Real-Time Monitoring Service (ARMS)—into a single, unified observability platform. Built around the proprietary UModel data model, it seamlessly integrates metrics, logs, and traces. This not only significantly reduces operational complexity but also provides a unified, inferable data foundation for upper-layer Agents.

Building upon CMS 2.0, Alibaba Cloud has introduced STAROps, a comprehensive AIOps platform designed with robust enterprise-grade safeguards, including RAM access control, manual approval for high-risk operations, Agent behavior auditing, and end-to-end encryption. With STAROps, enterprises can custom-build dedicated SRE Agents tailored to their specific needs, configuring distinct responsibilities, permissions, and skill sets so these Agents can autonomously schedule and execute IT operations tasks. Furthermore, STAROps functions as an overarching Agent running natively on Alibaba Cloud infrastructure, utilizing unified orchestration to handle complex, cross-domain operational workflows. Organizations can seamlessly integrate STAROps into their existing workflows via OpenAPI and MCPs to achieve immediate AI-driven efficiency gains, while simultaneously accelerating their upgrade to an Agent-native model for intelligent IT operations.

Alibaba Cloud's observability services now span the Asia-Pacific, Europe, the Middle East, Africa, and Latin America, with full support for OpenTelemetry data ingestion. Additionally, STAROps is now available on the Qoder Desktop plugin marketplace. By deeply integrating disparate cloud resources, observability data, and specialized domain expertise, STAROps is transforming IT operations from a reactive "firefighting" approach to a proactive, autonomous system—effectively bringing Agentic Ops into real-world production environments.

Source: Gartner, Magic Quadrant for Observability Platforms, By Padraig Byrne, Martin Caren, D.B. Cummings, Neil Young, 13th July 2026
Disclaimer:
Gartner does not endorse any company, vendor, product or service depicted in its publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner publications consist of the opinions of Gartner’s business and technology insights organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this publication, including any warranties of merchantability or fitness for a particular purpose. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Alibaba Cloud. Gartner and Magic Quadrant are trademarks of Gartner, Inc., and/or its affiliates.