STAROps Synthetic Monitoring Intelligent Analysis in Practice: From Task Analysis to Cross-Domain APM Fault Demarcation

This article explains how STAROps integrates Synthetic Monitoring, SLS, UModel, and APM to automatically analyze external access anomalies and accurately pinpoint cross-domain faults.

When a user reports that "the site won't load" or "the API suddenly got slow", network monitoring usually flags the anomaly quickly. The hard part comes next: did the fault occur on the network path between the user and the server, or inside the server-side application? Does it hit one region or ISP only, or every user? Should you call the network team, the application team, or the owner of a downstream dependency?

Synthetic Monitoring is the tool that watches this external access experience. Probe nodes spread across regions and ISPs follow fixed rules to simulate real users visiting a website, an API, or a domain name, and they record data for each stage: DNS resolution, TCP connection, TLS handshake, and server response. Traditional synthetic monitoring reports, however, are better at showing what happened than at explaining why it happened.

STAROps brings intelligent O&M analysis to observability data. Its Agent automatically queries task configuration, synthetic monitoring logs, and application call chains, then links evidence that used to sit in separate systems. In the end it answers four questions: where the anomaly occurred, which user paths it affected, which side owns the problem, and what data backs the conclusion.

This article is for product managers, developers, and O&M engineers who are new to Synthetic Monitoring intelligent analysis. It explains why STAROps connects Synthetic Monitoring, SLS, UModel, and APM, and how that chain moves you from spotting an anomaly to explaining it and pointing at a fix. By the end, you will understand where traditional synthetic monitoring analysis breaks down, how STAROps assembles evidence across systems, and how a network problem is told apart from a backend one.

Getting Started: Key Concepts in the Analysis Chain

• Synthetic Monitoring: Probe nodes actively visit a target website, API, or domain name and continuously check availability and performance from the user's point of view.
• STAROps Agent: The intelligent analysis unit that queries data, aggregates evidence, judges faults, and generates reports. Referred to as the Agent below.
• SLS: Alibaba Cloud Simple Log Service, which stores the raw execution record of every synthetic check. Think of it as the log warehouse you search for factual evidence during analysis.
• UModel: A unified model that describes observable objects such as tasks and applications, plus the relationships between them. Think of it as a map that tells the Agent who each object is, where its data lives, and how the objects connect.
• APM: Application Performance Monitoring, used to watch how a request executes after it reaches the backend. One complete request is a Trace, and each processing step within it is a Span.
• Workspace: An isolated scope of observability data and resources. The Agent must pin down the Workspace before it can locate the right task, logs, and APM data.

1. Background: Gaps in Traditional Synthetic Monitoring Analysis

Figure 1: Main gaps in traditional synthetic monitoring analysis

Traditional analysis usually starts with a synthetic monitoring report: check availability rate, response time, and error codes, then switch by hand between logs, task configuration, and APM pages to look for a cause. These metrics work as a health overview, but they rarely add up to verifiable root cause evidence, so four gaps remain in troubleshooting.

First, task context is missing. The same error code can mean completely different things under different task types, target addresses, DNS configurations, assertion rules, and monitoring point coverage. Repeating the error code alone tells you nothing about whether the cause is a misconfiguration, a broken target service, or a failure at one protocol stage.

Second, raw evidence is missing. OpenAPI is the standard query interface the system exposes, and central-side metrics are already aggregated statistics; both suit stable queries and health overviews, but neither carries all the detail an ad-hoc drill-down needs. Stage durations, city and ISP, target IP, CNAME, raw DNS answers, response headers, assertion results, and Trace context usually live only in the raw synthetic monitoring records in the user's SLS.

Third, protocol stages are not opened up. A single HTTP check may run through DNS, TCP, TLS, request send, first-packet wait, HTTP response, response download, and content assertion. Look only at total duration and the final error code, and problems from different layers blur together.

Fourth, synthetic monitoring and the backend application are not linked. Synthetic Monitoring can tell you that external access failed or slowed down, but once the request enters the backend, the problem may sit in business logic, the database, the cache, or a downstream dependency. In APM, the server span marks where the backend receives and handles the request, and child Spans record internal steps such as database, cache, and downstream calls. Without that evidence, the Agent cannot reliably pinpoint a problem inside the backend.

Synthetic Monitoring intelligent analysis therefore needs a complete chain: task identification → data location → raw sample query → field filtering and combined analysis → protocol-stage drill-down → link diagram rendering → cross-domain APM alignment → final fault demarcation.

2. Core Capabilities

Figure 2: The five core capabilities of STAROps Synthetic Monitoring intelligent analysis

1. Synthetic Monitoring Task Analysis

Every synthetic monitoring task carries a unique taskId. Once you supply a taskId, a task name, or a target address, the Agent locates the matching synthetics.task entity — the structured description UModel keeps for that task. It then reads the task type, target, probe frequency, monitoring points, timeout, assertions, and Trace configuration, and queries raw synthetic monitoring samples within the specified time range.

The analysis does more than compute sample count, success rate, availability rate, response time, and error distribution. It also filters, aggregates, and compares by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Even when a task looks stable overall, single-task analysis still checks protocol stages, localized slow points, and configuration risks instead of simply reporting "running normally".

Coverage today includes HTTP/HTTPS, Ping, TCP, UDP, DNS, DNSTRACE, SMTP, POP3, FTP, Traceroute, MTR, API, Multi, Browser, WebSocket, and SSL. Each task type follows its own analysis path: Ping focuses on packet loss and RTT, API and Multi on failed requests and steps, Browser on the main document, page timing, and the resource waterfall.

RTT (round-trip time) is the time data takes to travel from the probe to the target and back. API, Multi, and Browser correspond to API requests, multi-step tasks, and browser page loads respectively.

2. Intelligent Workspace Inspection

Single-task analysis answers "why is this task failing"; Workspace inspection answers "which tasks in this group deserve attention first". A Workspace is an isolated observability space that holds the tasks, logs, and related resources belonging to one user or business scope.

Workspace inspection first rolls up all synthetic monitoring tasks into an overall health view, then drills down into the tasks that matter. Task selection weighs status severity, failure count, availability rate, latency, error concentration, failing-stage concentration, and how long the anomaly has lasted, so the ranking never rests on a single metric.

Inspection supports three cycle scenarios:

• High-frequency monitoring: looks at the last 30 minutes for wide-scale unavailability, clustered timeouts, and high-priority risks, and analyzes the Top 1–3 tasks in depth.
• Daily diagnosis: analyzes the last 24 hours against the same window the previous day, and reports trends, lingering risks, and priority tasks.
• Weekly review: analyzes the last 7 days against the previous week, and identifies persistent anomalies, repeated fluctuations, recovered risks, and administration items.

3. Fine-Grained Protocol Diagnosis

The Agent picks protocol evidence to match the task type rather than explaining every task with one set of metrics:

• HTTP/HTTPS: analyzes stage by stage along DNS → TCP → TLS → request send → first-packet wait → HTTP response → response download → assertion.
• DNS: checks the query object, DNS Server, RCODE, CNAME, resolved IP, SOA, expected match, and whether the domain name exists.
• WebSocket: additionally checks whether the HTTP Upgrade completed.
• API and Multi: pinpoint the failing request or step.
• Browser: separates main-document, resource-loading, page-performance, and operation-step anomalies.

If network protocols are new to you: DNS translates a domain name into an IP address, TCP establishes a reliable connection, and TLS encrypts the traffic and validates the certificate. RCODE is the result status returned by the DNS server, CNAME marks a domain alias, and SOA records the basic information of the domain's authoritative zone.

Diagnosis covers configuration problems as well as data problems. A DNS query object mistakenly set to a full URL, an HTTP 200 whose content assertion fails, a disabled task, a query window shorter than the probe interval, or no matching samples in SLS — each is flagged on its own and never lumped in as a target service anomaly.

4. Request Link Visualization

The synthetic monitoring access link diagram is generated dynamically from real samples. The first node is always the probe, and only the protocol stages the request actually reached appear after it. If the request fails at DNS, the diagram stops at DNS and does not draw TCP, TLS, or the backend service.

The diagram follows a "stage node → anomaly node" structure. A DNS anomaly, for example, renders as "probe → DNS resolution → resolution anomaly (NXDOMAIN)" rather than jumping straight from the probe to the anomaly. Normal nodes, affected paths, and the final root cause use different styles, so an anomaly propagation path is not mistaken for the root cause.

Figure 3: Synthetic monitoring access link diagram generated from the actual execution stages in STAROps task analysis

The diagram does not depend on APM. With no Trace available, it still shows the synthetic monitoring access path; when a backend Trace exists, an APM call chain diagram is added to show the backend entry point, API, business processing, downstream dependencies, and the specific point of failure. The trace_id ties the two diagrams to the same request.

5. Cross-Domain APM Fault Demarcation

When an HTTP, API, Multi, or Browser task carries Trace evidence such as trace_id, traceparent, or traceInfo, the Agent picks a failed or high-latency sample and queries the APM Trace itself, instead of merely telling you to look it up.

A Trace is the full path of a request from the probe into the backend and on to other components; a Span is one step along that path. Linked analysis aligns four kinds of Span:

• Synthetic monitoring client span: the external view, from the probe sending the request to receiving the response.
• Backend server span: the entry view, from the request reaching the application to the application returning a response.
• Local business Span: the execution of a Handler, business method, or internal computation.
• Downstream dependency Span: a call to a database, cache, RPC, or external HTTP service.

By comparing synthetic monitoring stage durations, the server span, child Spans, and the first concrete error, the Agent can place the fault on the network path, the path before the application entry, the backend service, or a downstream dependency — or state that evidence is insufficient. A 502 from a backend API may be only the surface symptom; if the first concrete error appears in a downstream DNS, database, or external service Span, the conclusion moves to the downstream dependency.

3. Synthetic Monitoring Architecture and Technical Approach

Figure 4: Analysis architecture combining UModel, SLS, and APM

Data Organization: UModel and SLS Working in Two Layers

Think of UModel as the map and SLS as the field notes. UModel holds relatively stable object information: which task this is, what it targets, which Workspace it belongs to, where its data lives, and how it relates to other objects. SLS holds the execution facts that keep arriving: which probe hit which target at what time, which protocol stages it went through, and whether it succeeded.

Creating a synthetic monitoring task produces a synthetics.task entity in UModel that holds metadata such as taskId, task type, target address, probe configuration, Workspace, SLS data coordinates, and Trace configuration. Every real probe run then writes one or more execution records into SLS. One task entity therefore maps to many synthetic monitoring samples; each probe run does not create a new entity.

When a request arrives, the Agent first queries this map through UModelSearch to locate the task entity and its log location, then enters the matching SLS Project and Logstore to query samples by taskId and time range. A Project is the isolated resource space in SLS, and a Logstore is the data container inside it that holds one class of logs.

Analysis Strategy: Overall Aggregation, Risk Screening, and Protocol Drill-Down

With the samples in hand, the Agent first sizes up the scope and distribution of the anomaly, then drills down step by step into specific protocol stages. Single-task analysis aggregates by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Workspace inspection instead rolls up all tasks first, then screens out the ones with low availability rate, high latency, concentrated errors, or persistent anomalies.

After screening, the Agent reads representative failed or high-latency samples and takes a different analysis path per task type. An HTTP task, for instance, is checked in order for DNS, TCP, TLS, request send, first-packet wait, response download, HTTP status code, and content assertion; a DNS task goes on to read RCODE, the number of answer records, CNAME, resolved IP, SOA, DNS Server, and the match result.

This "overall aggregation → risk screening → protocol drill-down → representative sample" approach separates resolution failures, connection problems, certificate issues, slow server-side processing, and content assertion failures, instead of explaining only the final error code.

Cross-Domain Linkage: From the Synthetic Monitoring Domain into the APM Domain

The analysis chain can continue into APM only when the synthetic monitoring sample carries Trace information. For any task with Trace enabled, the system generates a synthetic APM service and uses the same_as relationship in UModel to map synthetics.task to that service permanently. For the Agent, this relationship is the signpost from the synthetic monitoring domain into the APM domain.

same_as answers "which synthetic APM object corresponds to this synthetic monitoring task"; trace_id answers "which applications and call nodes did this particular request pass through inside the backend". The first is a stable task-level relationship, the second a precise per-request association. Note one thing in particular: same_as does not equate the synthetic monitoring task with the real backend application.

Inside APM, the Agent looks at the synthetic monitoring client span, the backend server span, local business Spans, and downstream dependency Spans together:

• If synthetic monitoring duration rises and the server span rises with it, the problem leans toward the backend service.
• If most of the time sits in a database or downstream call, you can narrow it to that dependency.
• If the client span is slow while the server span is short, the problem more likely lies in the network, CDN, gateway, queuing, or another uninstrumented segment.
• If DNS, TCP, or TLS already failed and no server span exists, the request never reached the application.

If a sample has a trace_id but complete backend Spans are missing, the Agent explicitly marks "insufficient APM evidence" rather than reading "no data found" as "backend is normal".

Design Principle

The STAROps Agent does not hand a fixed report to a large model for summarizing. It builds a query plan first, then constrains its conclusions with real data. UModel serves as the semantic control plane that identifies objects and relationships, and the user-side Logstore serves as the data plane that holds observed facts: indexed fields screen samples fast, and SPL queries then fetch and process finer protocol details. Here, an index is a pre-built field for fast retrieval, and SPL is the query language used to filter, aggregate, and transform log data.

Different tasks load different evidence models, normalizing heterogeneous logs into five layers: task, single execution, protocol stage, blast radius, and call chain. When a sample carries Trace information, the Agent follows trace_id into APM and aligns the external client span with the backend server span, business Spans, and downstream dependency Spans. The output is not an empirical guess a large model made from field names; it is a traceable fault demarcation conclusion constrained by raw samples, aggregated results, protocol semantics, and cross-domain Traces.

End-to-End Analysis Chain

1. Parse the user input and decide whether the scope is a single task or a Workspace.
2. Query synthetics.task to obtain the task configuration and SLS data coordinates.
3. Fix the time window and calibrate time units.
4. Aggregate overall results and screen for risk using indexed fields.
5. For abnormal tasks, read representative samples and unindexed protocol fields.
6. Enter the dedicated analysis for DNS, HTTP, Browser, API, Multi, or another type as appropriate.
7. If a Trace exists, pick a failed or high-latency sample and query the APM call chain.
8. Align task configuration, synthetic monitoring stages, blast radius, and Span evidence, then output the fault demarcation conclusion, link diagram, recommended actions, and data limitations.

The core strategy is "overall aggregation → risk screening → protocol drill-down → representative sample → cross-domain alignment". It keeps query volume under control across long time windows and large Workspaces while preserving the raw evidence needed to pinpoint the failing stage and the root cause.

4. How to Use

For task analysis, simply supply a taskId, a task name, or a Workspace, for example:

Analyze the synthetic monitoring tasks over the last 30 minutes.

You do not have to write log queries yourself. The Agent turns a natural-language question into concrete steps: task location, sample query, protocol analysis, and APM linkage.

The Agent then completes the following steps automatically:

1. Identify the scope and confirm whether this is single-task analysis or Workspace inspection.
2. Find synthetics.task through UModelSearch or the task query capability.
3. Locate the user-side SLS Project and Logstore.
4. Query raw synthetic monitoring samples within the fixed time window.
5. Run protocol-stage analysis according to the task type.
6. If Trace evidence exists, go on to query the APM Trace.
7. Output the complete Synthetic Monitoring intelligent analysis report.

For ongoing inspection, choose the high-frequency monitoring, daily diagnosis, or weekly review template. The inspection scans the synthetic monitoring tasks in the Workspace on a schedule, screens out risky tasks, and generates a report automatically.

A "Long-Running Task" in STAROps is an analysis task that repeats automatically on a schedule; a "Digital Employee" is an Agent pre-configured with skills, tools, and data permissions. Combine the two and synthetic monitoring inspection runs on schedule and pushes its results to the people you designate.

Synthetic Monitoring Task Analysis

Figure 5: Starting a synthetic monitoring task analysis in the STAROps console

In the STAROps console, you can ask about synthetic monitoring task details directly, for example "Analyze the synthetic monitoring tasks over the last half hour", "Analyze why a given synthetic monitoring task is failing", or "Show the synthetic monitoring tasks whose availability rate dropped sharply in the last day".

Synthetic Monitoring Inspection Tasks

Figure 6: Opening Long-Running Task in STAROps

1. Click "Long-Running Task" in the upper-left corner of the STAROps console.

Figure 7: Creating a synthetic monitoring inspection task and configuring the Workspace

1. Click "Create Task" in the upper-right corner of the page, select a Digital Employee and the Workspace the task belongs to, and enter a prompt such as "synthetic monitoring inspection".
2. Select one of the three synthetic monitoring inspection templates or combine them, and describe the inspection details in text; by default the inspection runs at Workspace level.

Figure 8: Configuring notification recipients and adding the job to Long-Running Task

4.  Click "Configure Now" to add notification recipients, then click "Add to Long-Running Task" once you have made your selection.

5.  Confirm the configuration is correct, then click "Confirm Execution".

Figure 9: Viewing the inspection report generated by the Long-Running Task

6.  Click "Report" at the top of the page to view the detailed inspection results.

5. Fault Demarcation Logic

1. Locate the Layer Where the Request Stopped

An HTTP call chain runs DNS → TCP → TLS → request send → first-packet wait → HTTP response → download → assertion. The deepest stage the Agent can observe tells it whether the request reached the service entry point.

A DNS failure with no TCP evidence, for example, means the request never established a connection. An HTTP 502 shows that DNS resolution, the TCP handshake, and the response header return all completed, so the problem cannot be a first-packet wait timeout. An HTTP 200 with a failed assertion means the service is reachable and the failure sits in the response content or the business logic.

2. Determine the Scope of Impact and Temporal Pattern

Simultaneous failures across multiple cities and ISPs point to the target service, a public entry point, or a cross-network path. Concentrated failures in a single city or ISP suggest a regional network or ISP link issue. If only some monitoring points fail within the same city and ISP, the Agent drills down by client_id, DNS Server, local DNS, and egress path when these fields are available.

The Agent also separates continuous, sudden, intermittent, and recovered anomalies over time. It never generalizes a single spike into a continuous failure, and averages never stand in for P95, maximum values, or anomalous time buckets.

3. Prioritize DNS Evidence Over Outer Error Codes

The DNS error_code reflects the probe's local judgment, while RCODE is the protocol-layer response from the DNS server — the two are not interchangeable.

When raw_exception is present, the Agent reads rcode, rcode_name, and the answer record count first, to tell NXDOMAIN, SERVFAIL, REFUSED, FORMERR, NOTIMP, and NODATA apart (NODATA means RCODE is NOERROR but the target record is empty). If those fields are missing, it draws evidence from dns_raw, dns_raw_inner, message, SOA, EDE, and DNSSEC information, and states the confidence level of the conclusion explicitly.

DNS analysis also compares the DNS Server, city, ISP, monitoring point, CNAME, resolved IP, and target edge zone. Note that if error_code=0 but only an SOA record is returned without A, AAAA, or CNAME records, this only indicates that the current rule evaluation passed; it does not prove that the target record resolved normally.

4. APM Evidence Determines the Backend Boundary

• If total synthetic monitoring duration rises while DNS, TCP, and TLS stay normal, and the server span rises with it, the problem lies in the backend service.
• If a specific business sub-Span within the server span accounts for most of the duration, the application's internal processing is slow.
• If the first concrete error, or most of the duration, appears in a downstream Dependency Span, the problem lies in that downstream dependency.
• If the client span is markedly slow while the server span is very short, the problem more likely lies in the path before the application entry, the gateway, the CDN, or an uninstrumented component.
• If DNS, TCP, or TLS fails and there is no server span, the request never entered the application.
• If a trace_id exists but APM has no data or the Spans are incomplete, the evidence is insufficient, and you cannot conclude from it that the backend is healthy.

The gap between the client span and the server span is not pure network time; it may also include time spent in the CDN, gateway, ingress proxy, queueing, and uninstrumented components. The final conclusion therefore has to draw on synthetic monitoring stage fields, target access evidence, and Span parent-child relationships together.

6. How Cross-Domain APM Analysis Works

Figure 10: Analysis flow from a synthetic monitoring task into the APM call chain

A full cross-domain APM analysis takes four steps:

1. Starting from the synthetic monitoring task, the Agent uses UModelSearch to locate the synthetics.task entity, confirming the task type, target address, Workspace, and data coordinates.
2. The Agent queries the synthetic monitoring SLS samples to complete the analysis on the synthetic monitoring side; if the samples carry Trace evidence, it extracts a representative trace_id.
3. The Agent uses the trace_id to query the APM Trace, retrieving the server span, business processing Span, and downstream Dependency Span for the same request within the backend application.
4. The Agent aligns the evidence from the synthetic monitoring side and the APM side to form a cross-domain fault demarcation conclusion.

Evidence alignment follows these rules:

• Synthetic monitoring is slow and the APM server span slows with it: the problem most likely lies in the backend service.
• Synthetic monitoring is slow and the APM downstream Dependency Span slows with it: the problem most likely lies in a backend downstream dependency.
• Synthetic monitoring is slow but the APM server span is very short: the problem leans toward the network link, the ingress layer, the gateway, or the path before the application entry.
• The DNS, TCP, or TLS stage fails and there is no server span: the request never entered the application, so look first at the path before the application entry.
• Trace evidence exists but no backend server span can be found: you cannot conclude that the backend is healthy. The Agent can only flag "insufficient APM evidence" and check Trace reporting, sampling, Region, Workspace, and APM integration.

The APM linkage presents evidence as two link diagrams: one shows the external synthetic monitoring access path, the other the backend Trace path, and trace_id aligns them as the same request. Together they show whether the request reached the service entry point and, once it was inside the application, exactly which Span failed.

Figure 11: Aligning the synthetic monitoring access path with the backend APM Trace via trace_id

7. From "Seeing the Anomaly" to "Knowing What to Do Next"

The value of STAROps is not longer reports; it is a shorter distance from alert to action. It organizes task configuration, raw SLS samples, monitoring point distribution, protocol stages, and APM Traces into one continuous chain of evidence. Users can start from a taskId, task name, or Workspace and work out step by step whether an anomaly comes from a global outage, a regional network, a single ISP, a specific monitoring point, the task configuration, or the backend application and its downstream dependencies.

For experienced engineers, these diagnostic capabilities cut down the constant switching between the Synthetic Monitoring console, SLS, and APM to piece evidence together by hand. For anyone handling a synthetic monitoring fault for the first time, they turn an expert's troubleshooting sequence into a repeatable analysis flow. At Workspace level, the Agent can also pick high-risk targets out of a large pool of tasks and keep running high-frequency monitoring, daily diagnosis, and weekly reviews.

Appendix: Glossary of Terms

Term	Definition
Synthetic Monitoring	Simulates user access from distributed probe nodes to proactively check the availability and performance of target services.
STAROps Agent	An intelligent analysis unit responsible for invoking tools, querying data, aggregating evidence, and generating diagnostic conclusions.
Workspace	A mutually isolated set of observable resources and data scopes.
taskId	The ID that uniquely identifies a synthetic monitoring task.
UModel	A unified model describing observable entities and their relationships, used to locate analysis targets and navigate across domains.
SLS	Alibaba Cloud Simple Log Service (SLS), used to store and query synthetic monitoring execution samples and related logs.
Project / Logstore	A Project is a resource isolation space in SLS, and a Logstore is a data container within it that stores a specific category of logs.
SPL	A query language used to filter, aggregate, and transform log data.
synthetics.task	The entity type in UModel that represents a synthetic monitoring task.
APM	Application Performance Monitoring, used to observe the execution of requests after they enter the backend application.
Trace / Span	A Trace is the complete call chain of a single request, and a Span is an individual processing step within that chain.
same_as	The UModel relationship that associates a synthetic monitoring task entity with its corresponding synthetic monitoring APM service.

Conclusion

Synthetic Monitoring captures the external user experience before a request reaches the service, APM observes how the request executes once inside the application, SLS preserves the factual evidence, and UModel locates the objects and connects the different data domains. STAROps organizes these four into a continuous chain that runs from task discovery through sample aggregation and protocol diagnosis to backend fault demarcation.

This practice turns Synthetic Monitoring from a system that only displays abnormal metrics into an analysis entry point that explains anomalies and supports decisions. Instead of a raw list of fields, users get verifiable answers: where the request stopped, who the anomaly affected, which side owns the fault, what the evidence is, and what to tackle first.