DEV Community: ObservabilityGuy

Say Goodbye to Complex Onboarding: AI Agent Skills Drive Cloud Monitor's Observability Integration

ObservabilityGuy — Thu, 13 Aug 2026 07:04:32 +0000

This article introduces Alibaba Cloud Monitor's AI Agent Skill, which automates complex observability integration through simple natural language commands.

Background

With the popularization of cloud-native architecture and the rapid growth of AI applications, the types of applications enterprises need to manage are becoming increasingly diverse—from traditional Java microservices to AI Agents, and from Go backends to various AI gateway components. Meanwhile, configuring the integration for observability platforms involves numerous parameters and steps, placing higher demands on operational efficiency.

As the unified observability management platform for Alibaba Cloud, Cloud Monitor (CMS) 2.0 integrates core capabilities such as Application Monitoring (application performance management, APM), Real User Monitoring (RUM), Managed Service for Prometheus, and Alert Management. To enable users to efficiently complete observability integration in terminal environments, the CMS team introduced the aliyun cms2 CLI tool. Taking it a step further, by encapsulating CLI capabilities into the alibabacloud-cms-manage Skill, we have achieved intelligent observability integration based on AI Agents. Users simply describe their requirements in natural language, and the AI Agent automatically orchestrates CLI commands to complete the entire process.

This article will introduce the application integration capabilities of the CMS CLI, with a focus on demonstrating how to achieve automated integration in K8s scenarios using the AI Agent Skill.

Figure 1: CMS CLI + AI Agent Skill

CMS CLI Overview

The aliyun cms2 is a subcommand plugin for the Alibaba Cloud CLI that covers command-line operations for all modules in CMS 2.0:

Ensure your environment is ready before use:

# Confirm the CLI is installed and the version is >= 3.3.15 
$ aliyun version 
3.3.20

# Verify the cms2 plugin is available
$ aliyun cms2 --help

# Configure credentials (if not configured)
$ aliyun configure

Application Integration Capabilities

Application Monitoring and AI Observability

The Application Monitoring module of the CMS CLI supports application integration for multiple languages and provides three integration methods: ack-onepilot (for K8s containers), manual proprietary agent, and native OpenTelemetry:

For AI Observability, CMS 2.0 provides an out-of-the-box integration experience for mainstream AI frameworks:

CLI Integration Workflow
Regardless of the application's language or integration method, CLI integration follows these 6 steps:

Figure 2: 6-step CLI integration workflow

Below are the core commands for each step:

# Step 1: Get the account ID 
$ aliyun sts get-caller-identity --force -o json 
# → AccountId: 1108xxxxxxxxxxxx

# Step 2: Initialize Application Monitoring infrastructure (idempotent)
$ aliyun cms2 apm configuration create \
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou \
    --region cn-hangzhou

# Step 3: Get access credentials (license key, endpoint, etc.)
$ aliyun cms2 apm configuration get \
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou \
    --region cn-hangzhou -o json

{
  "success": true,
  "data": {
    "entryPointInfo": {
      "authToken": "a]***@***************4b70",
      "privateDomain": "proj-xtrace-***-cn-hangzhou.cn-hangzhou-intranet.log.aliyuncs.com",
      "project": "proj-xtrace-***-cn-hangzhou",
      "publicDomain": "proj-xtrace-***-cn-hangzhou.cn-hangzhou.log.aliyuncs.com"
    },
    "status": "Running",
    "workspace": "default-cms-1108xxxxxxxxxxxx-cn-hangzhou"
  }
}

# Step 4: Register the application service 
$ aliyun cms2 apm service create \ 
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou \ 
    --region cn-hangzhou \ 
    --body '{"serviceName":"my-app","serviceType":"TRACE","attributes":"{\"language\":\"java\"}"}' \ 
    < /dev/null 
# → serviceId: a]***@*********************7f1

# Step 5: Get the integration configuration template (using Java OTel as an example)
$ aliyun cms2 integration addon get --addon-name apm-java-batch --env-type Client -o json

# Step 6: Verify the integration
$ aliyun cms2 apm service list \
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou \
    --service-name my-app --region cn-hangzhou

Although the above workflow is clear, executing 6 steps and passing multiple parameters can still present a learning curve for O&M engineers who do not use the CLI frequently. Next, we will introduce how to simplify this entire process into a single natural language sentence using the AI Agent Skill.

Figure 3: Easy integration with a single sentence using the Skill

Achieving Intelligent Integration via AI Agent Skills

alibabacloud-cms-manage Skill

The CMS team has encapsulated the complete CLI operational knowledge into an out-of-the-box Skill: alibabacloud-cms-manage. Its core concept is to transform the CLI operational workflow into a structured workflow executable by the AI Agent. Users do not need to memorize commands and parameters; they simply describe their needs in natural language to complete the integration.

The Skill covers the following modules:

For details on installing and using the Skill, see https://skills.alibabacloud.com/skills/alibabacloud-cms-manage.

Working Mechanism
When a user describes their requirements in Claude Code, the AI Agent operates according to the following workflow:

Figure 4: Intelligent Skill integration workflow

Security Mechanism: Two-Phase Confirmation Protocol
For operations that may affect the user's cluster (such as installing components or deploying patches), the Skill strictly enforces a two-phase confirmation:

Read-only commands (get, list) and CMS backend resource creation (apm configuration create) do not require confirmation and can be executed directly by the Agent.

Figure 5: Two-phase confirmation protocol

Demonstration: Automatic Integration for K8s Containers
Below is the complete interaction process for using this Skill to automaticallyintegrate a LangChain application into a Container Service for Kubernetes (ACK) cluster. The ack-onepilot runs an Agent Pod on each cluster node via a DaemonSet. Once a Deployment is annotated with the specified label, ack-onepilot automatically injects the agent — no changes to the application code or Dockerfile are required.

User:

Help me integrate the LangChain application customer-support-agent in my ACK cluster into CMS.

AI Agent：

Detected a container integration scenario. Fetching cluster information...

$ aliyun sts get-caller-identity --force -o json
# → AccountId: 1108xxxxxxxxxxxx

$ aliyun cs describe-clusters
# → clusterId: c8d3f***********, region_id: cn-hangzhou

Initializing APM infrastructure and obtaining credentials...

$ aliyun cms2 apm configuration create \
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou --region cn-hangzhou
# → 成功

$ aliyun cms2 apm configuration get \
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou --region cn-hangzhou -o json
# → LicenseKey: a]***@***************4b70

$ aliyun cms2 apm service create \
    --workspace default-cms-1108xxxxxxxxxxxx-cn-hangzhou --region cn-hangzhou \
    --body '{"serviceName":"customer-support-agent","serviceType":"TRACE","attributes":"{\"language\":\"python\"}"}' \
    < /dev/null
# → serviceId: a]***@*********************7f1

Checking the status of the ack-onepilot component...

$ kubectl get pods -n ack-onepilot
# NAME                      READY   STATUS    RESTARTS   AGE
# ack-onepilot-7d8f9        1/1     Running   0          5d

The ack-onepilot is installed and running normally. Searching for the target Deployment...

$ kubectl get deployment --all-namespaces | grep customer-support
# default   customer-support-agent   3/3     3    3    12d

Found the target Deployment default/customer-support-agent (3 replicas). Execution plan — Patch K8s Deployment. Please confirm the execution (yes/no).

User:

yes

AI Agent：

Executing the patch...

$ kubectl patch deployment customer-support-agent -n default \
    --type=strategic -p '{
      "spec":{"template":{"metadata":{"labels":{
        "aliyun.com/app-language":"python",
        "armsPilotAutoEnable":"on",
        "armsPilotCreateAppName":"customer-support-agent",
        "armsPilotAppWorkspace":"default-cms-1108xxxxxxxxxxxx-cn-hangzhou"
      }}}}}'
# deployment.apps/customer-support-agent patched

Verifying the rolling update...

$ kubectl rollout status deployment/customer-support-agent -n default --timeout=120s 
# deployment "customer-support-agent" successfully rolled out

Integration completed. The application has had the Python agent automatically injected via ack-onepilot. In 2-3 minutes, you will be able to view monitoring data such as LLM invocation latency, Token usage, and Agent traces in the CMS console.

Throughout this process, the user needed only to provide a single natural language description. The AI Agent automatically handled account retrieval, cluster information derivation (eliminating the need to manually supply a region ID), infrastructure initialization, credential retrieval, service registration, component status checks, and Deployment lookups — while also applying a two-phase confirmationmechanism that lets the user review the patch content before execution. Compared with the 8+ commands required for manual operation, this Skill distills the integration experience — from memorizing commands, looking up parameters, and assembling JSON — down to a single sentence.

Figure 6: K8s ack-onepilot automatic integration

If you are interested in the CMS CLI or AI Agent-driven observability integration, check out the following resources:

● alibabacloud-cms-manage Skill

● CMS 2.0 Console

● CMS documentation

● Alibaba Cloud CLI installation guide

From Individual Productivity to Organization Capability: AI Coding Measure Practices of LoongSuite-Pilot and SLS

ObservabilityGuy — Thu, 13 Aug 2026 06:31:18 +0000

This article details an engineering approach using LoongSuite-Pilot and Alibaba Cloud SLS to measure, analyze, and optimize organizational AI coding agent usage and ROI.

Introduction: Coding Has Become Faster, But Why Has the Organization Not Kept Up?

In May 2026, the Google Cloud DORA team published ROI of AI-Assisted Software Development. Unlike DORA Accelerate State of DevOps Report 2025 from the previous year, which focused on individual adoption rates, this report directly addresses an organization-level issue:

"The question is no longer whether AI-assisted development works — it's how to prove it to the business."

The report provides a group of model data. Taking an engineering team of 500 staffs as an example, the annual investment in AI tools is approximately $8.4 million, the expected return is $11.6 million, and the first-year ROI is approximately 39%. However, this return is not automatically realized. DORA explicitly points out that the improvement in coding speed does not automatically result in conversion into organization output:

"Initial gains in coding speed are promising, but they don't automatically translate to the bottom line."

The report introduces the concept of the J-Curve (a curve that first drops and then rises) to explain the mechanism. The team will experience a productivity decline period in the early stage of adopting AI tools. Workflow adaptation, habit switch, and prompt tuning are all learning costs. Only after the team passes this trough and reinvests the recovered capacity to reduce rework instead of directly cutting manpower, will the ROI be realized in the later segment of the curve. At the same time, DORA continues the core judgment of the 2025 report:

"AI is an amplifier, not a transformer. It magnifies strengths and dysfunctions."

Data shows that AI can bring an efficiency gain of 35% to 40% on greenfield projects, but less than 10% on legacy code. Such a huge difference means that the return curves of different teams within the same company may be completely different. An organization without event-level measure capabilities cannot even judge where its own J-Curve has reached.

This is exactly the dilemma of most current research and development organizations: they only have two classes of data in hand. One is the self-reported satisfaction questionnaire of individuals (subjective and untraceable), and the other is the aggregation KPI of the CI/CD pipeline (which tells you what, but does not explain why). What is truly missing is the middle layer: the event-level AI coding usage behavior measure that can drill down to the Agent, model, skill, or department. This allows the organization to accurately answer where the J-Curve has reached, which teams are already in the later segment of the curve, and which are still in the trough.

The LoongSuite-Pilot × Alibaba Cloud Simple Log Service (SLS) integration introduced in this topic is exactly the engineering implementation of this measure layer. LoongSuite-Pilot uniformly collects the event streams of heterogeneous Agents according to the LoongSuite GenAI semantics specification (an extension specification launched by Alibaba Cloud based on the OpenTelemetry GenAI semantic conventions). The SLS dashboard interprets the events into organization-level measures that can be used to drill down, perform attribution, and take actions.

Data Ingestion Layer: from Collection to SLS Implementation

Before you build the measure dashboard, you can first solve the problem of where the data comes from.

What we choose to measure is not how many lines of code are committed or how many pull requests (PRs) are merged, but the usage behavior of the AI coding Agent itself: who uses which Agent, which model is selected, how many tokens are consumed, and which tools and skills are invoked.

The data model adopts the classic separation design of the fact table and the dimension table. The two tables are associated through user.id and work_no. The separation of the fact table and the dimension table allows the measure layer to flexibly superimpose organization dimensions through JOIN, without the need to embed organization information during event reporting.

Event Fact Table: Behavior Logs of the AI Coding Agent
The fact table is the core data source of the entire measure dashboard. Each record corresponds to one Agent invoke event. The core fields include:

User identifier: user.id.

Session: gen_ai.session.id.

Agent/Provider/Model: gen_ai.agent.type, gen_ai.provider.name, gen_ai.request.model, and gen_ai.response.model.

Token consumption: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.usage.total_tokens.

Tool calling: gen_ai.tool.name, gen_ai.tool.call.arguments.file_path, and event.name.

These fields align with the LoongSuite GenAI semantics specification. OpenTelemetry GenAI semantic conventions are recognized by the industry as a starting point. However, community standards naturally need to balance broad applicability and long-term stability, and are currently still in the development status. In actual business, the invocation chain of AI Agents is often much more complex than "single user × single model." A request may span the collaborative invocation of multiple agents.

The LoongSuite GenAI semantics specification is exactly an extension specification derived from a large number of practical scenarios based on the OpenTelemetry community standards. The specification is officially open-source now, and the optimization capabilities will be gradually contributed to the community upstream in the future. The direct benefit of aligning with this specification is that the reporting metrics of Claude Code, Copilot, Cursor, Qoder, and various internal Agents are naturally consistent. You do not need to perform field reconciliation afterward.

These events are uniformly collected through LoongSuite-Pilot. LoongSuite-Pilot does not distinguish between agent sources and integrated development environment (IDE) forms. Whether it is Claude Code in the command line form or Copilot or Cursor in the IDE plugin form, they all fall into the same fact table. The collection granularity is at the event level. Each agent invocation, each tool call, and each token consumption are reported as independent events, rather than being reported after being aggregated by session.

The event-level granularity brings three engineering values:

First, comparability of heterogeneous agents: The statistics metric of the same token under Claude Code and Copilot is completely consistent. Cross-tool comparison and aggregation do not require any metric conversion.

Second, session-level traceability: By drilling from gen_ai.session.id to a single tool.call, troubleshooting can be accurate to "what parameters were passed in the third tool calling in this session," rather than stopping at "this person had a high token consumption last week."

Third, observability of skills and tools: gen_ai.tool.call.arguments.file_path associates the tool calling with the specific SKILL.md file path, making "whether the skills accumulated by the team are really used" a quantifiable problem.

Personnel Dimension Table: Organization Relationship Mapping
The event stream resolves "what happened," and the personnel dimension table resolves "who is doing it." The user identifier in an event is mapped to the organization topology, so that the measure can drill down by department, team, and personnel. The dimension table is periodically synchronized to SLS. The core fields are work_no (employee ID), show_name (name), and dept_name (full department path, such as Technology R&D Department - Engineering Platform Department – Data Service group).

Why must the organization relationship be integrated as an independent dimension table, rather than injecting department information into the event when the event is reported? There are three reasons:

First, decoupling. The personnel dimension table is continuously updated with organization adjustments, but once an event is reported, it should not be retroactively modified. Only when the dimension table is independent can you "view historical data based on the latest organization structure."

Second, LEFT JOIN exposes blanks. For employees who have records in the dimension table but no records in the event table, you can directly list the people who are "registered but not reported" through LEFT JOIN + WHERE user_id IS NULL. This list of "registered but not reported" people can often drive implementation better than all cool charts.

Third, level splitting is completed at one time. The department path is split by level into three layers: tier-1 department, tier-2 department, and team. All downstream charts only need to JOIN the dimension table to obtain the complete organization dimension. You do not need to repeat the splitting logic for each graph.

The core questions answered by the data integration layer boil down to three: what to collect (LoongSuite GenAI semantics events), how to collect (uniform collection by LoongSuite-Pilot), and who to associate with (personnel dimension table). After the three steps are completed, SLS has a complete data foundation of "event × organization."

Measure Layer: from a Public CTE to an Actionable AI Coding Agent Measure Dashboard

Why Choose the SLS Dashboard as the Analysis Layer?

After having the data foundation, the next step is to choose an analysis carrier. We directly build the entire measure dashboard on the SLS dashboard using SQL. The core reason for choosing this path is flexibility. AI coding measure is a scenario where the metric highly varies by team and requirements undergo fast iteration. The analysis layer must give users sufficient freedom.

First, query is definition. Behind each chart on an SLS dashboard is an SQL statement. If requirements change, you can modify the SQL statement to make the changes take effect immediately. You do not need to wait for the product to be published or for configuration support.

Second, definitions are fully self-controlled. "What counts as an active user", "how to break down departments", and "whether the denominator for the coverage rate is registered employees or all employees" — these definition choices vary by organization and have no ground truth. SLS SQL allows each team to flexibly define them based on their own business characteristics. Your organization can break down by level-3 departments, while mine breaks down by project groups. Your definition of active is "having events in the past 7 days", while mine is "having events in the past 30 days and Tokens > 1000". These differences can be expressed by a single WHERE clause at the SQL layer.

Third, collection and analysis are on the same platform. After events are delivered to SLS, they can be queried immediately. There is no additional extract, transform, and load (ETL) link or T+1 delay. For example, if you just connect a new Agent, you can confirm whether data flows in normally on the dashboard within a few minutes after it is published. The verification closed loop is extremely short.

After flexibility is established, the next question is: How to maintain definition consistency across 30+ charts?

CTE: the Engineering Skeleton of a Report

Flexibility solves the problem of "whether it can be modified". However, if each chart on a dashboard with 30+ charts has its own SQL statement, definition inconsistencies and maintenance costs will quickly spiral out of control. Therefore, the engineering core lies in the design of the analysis hierarchy: from metric definition, to pre-aggregation, to dimension progression, and then to specific charts, each layer serves the next layer.

All charts on the entire dashboard share the same group of CTE. This is not an SQL trick, but the engineering prerequisite for the entire report to achieve "definition consistency, maintainability, and controllable performance".

CTE 1: dept_user (Standardization of Personnel Dimension Table)

WITH dept_user AS (
SELECT
work_no,
show_name,
COALESCE(SPLIT_PART(dept_name, '-', 1), '') AS dept_name_1, -- Level-1 department, such as "Technology R&D Department"
COALESCE(SPLIT_PART(dept_name, '-', 2), '') AS dept_name_2, -- Level-2 department, such as "Engineering Platform Department"
COALESCE(SPLIT_PART(dept_name, '-', 3), '') AS dept_name_3 -- Team, such as "Data Service group"
FROM <dept-logstore>
GROUP BY work_no, show_name, dept_name
)

The core of this CTE is to break down the complete department path (such as Technology R&D Department-Engineering Platform Department-Data Service group) into a three-level hierarchy (level-1 department/level-2 department/team). The definition constraint of "statistics scope" is defined only once here. All downstream charts automatically inherit it through JOIN.

CTE 2: active_user (Event Pre-aggregation)

active_user AS (
 SELECT
 date_trunc('day', __time__) AS t,
 "user.id" AS user_id,
 coalesce(nullif("gen_ai.agent.type", 'null'), 'unknown') AS agent_type,
 coalesce(nullif("gen_ai.provider.name", 'null'), 'unknown') AS provider,
 coalesce(nullif("gen_ai.request.model", 'null'),
 nullif("gen_ai.response.model", 'null'), 'unknown') AS model,
 sum(coalesce("gen_ai.usage.input_tokens", 0)) AS input_tokens,
 sum(coalesce("gen_ai.usage.output_tokens", 0)) AS output_tokens,
 sum(coalesce("gen_ai.usage.total_tokens", 0)) AS total_tokens,
 count(1) AS events
 FROM <events-logstore>
 GROUP BY t, user_id, agent_type, provider, model
)

This CTE performs five-dimension pre-aggregation by day × user × Agent × supplier × model to produce the number of Tokens and events. The key benefit of pre-aggregation is: The vast majority of charts directly reuse active_user JOIN dept_user, and you do not need to repeatedly write aggregation logic for each chart.

Convention for JOIN between two tables:

FROM dept_user d
JOIN active_user a ON d.work_no = a.user_id

The design quality of the CTE layer determines three fundamentals of the entire dashboard: definition consistency (constraints are defined only once), maintainability (modifying one CTE automatically takes effect for all charts), and query performance (pre-aggregation reduces the scan volume of downstream SQL statements). For the 8 sections detailed later, the vast majority directly reuse CTE as the data source. A small number of charts that require finer granularity (such as Skill path fetch and repository dimension aggregation) revert to the original event table for separate queries. However, even so, the JOIN logic of the personnel dimension table still reuses dept_user to maintain definition consistency.

Analysis Dimension Progression
The 8 sections of the measure dashboard are not randomly stacked charts, but are designed progressively according to the analysis hierarchy:

From "overall perception" to "structure breakdown" and then to "threat validation", each layer provides context for finer-granularity analysis.

The data in the following screenshots is all analog data, which is used to illustrate the report structure and analysis logic, and does not represent real business data.

Section 1: Overview

A row of comparison cards at the top serves as the 'water level gauge' for the entire dashboard: active employees, total tokens, session count, Agent events, tokens per capita, and employees without reported usage. Each card includes a week-over-week (WoW) comparison against the same period last week. Together, these cards elevate the question of "how much is being used" from individual perception to the organizational level—revealing how many people are actively using it and whether overall usage this week is trending up or down compared to last week.

Cards showing abnormal week-over-week changes serve as drill-down entry points: if a metric drops, you can navigate to the corresponding section to identify the root cause. This shifts management decisions from intuition-driven to evidence-based.

To ensure WoW figures are reliable, the WoW calculation needs to work around a common pitfall: compare() is not compatible with cross-table CTE joins, and will throw an error directly. To address this, the calculation is instead implemented using manual windowing:

time_base AS (
  SELECT max(__time__) AS t_max FROM <event table>
),
cur_agg AS (
  SELECT "user.id" AS user_id, count(*) AS events
  FROM <event table>
  WHERE __time__ >= (SELECT t_max - 604800 FROM time_base)
  GROUP BY "user.id"
),
prev_agg AS (
  SELECT "user.id" AS user_id, count(*) AS events
  FROM <event table>
  WHERE __time__ >= (SELECT t_max - 1209600 FROM time_base)
    AND __time__ < (SELECT t_max - 604800 FROM time_base)
  GROUP BY "user.id"
)
-- JOIN dept_user separately and compute the WoW comparison after aggregation
SELECT
  cur.cnt AS "Current value",
  CASE WHEN prev.cnt > 0
       THEN round((1.0 × cur.cnt / prev.cnt - 1) × 100, 2)
  END AS "Compared to last week (%)"
FROM cur, prev

The SQL is longer, but it's the only reliably stable syntax under the "CTE + JOIN" architecture.

Section 2: Structure Distribution

Three pie charts segment the token proportion by agent_type, model, and provider respectively, answering the resource configuration questions at the organization layer: which Agent is taking the lead, whether tokens are concentrated in one supplier or scattered among multiple suppliers, and what the main model of the team is. The answers directly determine whether the tools require convergence, how suppliers are centrally procured, and how costs are aggregated.

The data required for these decisions becomes very lightweight to obtain thanks to the CTE base: all three graphs run on dept_user JOIN active_user, and only the GROUP BY fields are different:

SELECT
  a.agent_type AS "AI Agent",
  sum(a.total_tokens) AS "Total tokens"
FROM dept_user d
JOIN active_user a ON d.work_no = a.user_id
GROUP BY a.agent_type
ORDER BY "Total tokens" DESC

Pre-aggregation allows each chart to focus solely on its own GROUP BY dimension, and switching model or provider versions is simply a matter of changing a single field.

Section 3: Trend

Trend charts extend the "current status" to the "trend": the coverage size trend (a double line for the number of employees and the number of events), the Token consumption trend (three lines for input/output/total), the event and Token trends of each Agent (grouped by agent_type), and the distribution of Token usage time periods.

You can only make judgments from trends on whether the usage size of AI tools is steadily growing or falling back after a wave of promotion, and whether a specific Agent is becoming increasingly important or gradually overridden. The time period distribution provides a profile of the work rhythm. If Tokens are concentrated in the early morning, it indicates that the proportion of AI automation jobs is increasing. This is a completely different pattern from the usage driven by manual interaction during working hours, and the management actions are also different.

Most line charts can directly reuse the day granularity GROUP BY a.t of active_user. However, the time period distribution is an exception. The active_user has already been processed with date_trunc('day', __time__), and the hour dimension is discarded. To revert the hourly distribution, you must bypass the CTE and start a new hourly_user for independent aggregation:

hourly_user AS (
  SELECT
    date_format(__time__, '%H:00') AS h,
    "user.id" AS user_id,
    sum(coalesce("gen_ai.usage.total_tokens", 0)) AS total_tokens
  FROM < event table>
  GROUP BY h, user_id
)
SELECT a.h AS t, sum(a.total_tokens) AS "tokens"
FROM dept_user d
JOIN hourly_user a ON d.work_no = a.user_id
GROUP BY a.h
ORDER BY t

This is a typical backoff path when the downstream chart requires a finer granularity than the CTE.

Section 4: Department Statistics

Department statistics shift the perspective from individuals to teams. The department Token fact table (department, total number of people, number of users, coverage rate %, number of events, input/output/total Tokens, and Tokens per capita) is paired with the "list of employees who did not normally upload statistics data". Then, a Top 10 horizontal bar chart is overlaid, with sorting applied by total Tokens, coverage rate, and Tokens per capita, respectively.

The three dimensions point to three different actions: departments with a low size need promotion, departments with a low coverage rate need to be advanced from "individual early adoption" to "team standard", and departments with a low per capita usage may require optimization of tool configurations or improvement in usage methods.

To make these rankings meaningful, departments with "0 users" cannot be filtered out, because they are exactly the targets of promotion. Therefore, the fact table uses LEFT JOIN instead of INNER JOIN:

SELECT
  d.dept_name_2 AS "Department",
  approx_distinct(d.work_no) AS "Total number of people",
  approx_distinct(a.user_id) AS "Number of users",
  round(100.0 × approx_distinct(a.user_id) / nullif(approx_distinct(d.work_no), 0), 2) AS "Coverage rate (%)",
  sum(a.total_tokens) AS "Total Tokens",
  round(1.0 × sum(a.total_tokens) / nullif(approx_distinct(a.user_id), 0), 2) AS "Tokens per capita"
FROM dept_user d
LEFT JOIN active_user a ON d.work_no = a.user_id
GROUP BY d.dept_name_2
ORDER BY "Total Tokens" DESC

LEFT JOIN allows departments with "0 users" to be retained in the table. These departments will not be silently filtered out by INNER JOIN. The "unreported" list is the same JOIN with the addition of WHERE a.user_id IS NULL. A single JOIN simultaneously supports two classes of outputs: "size ranking" and "blank search". The latter often drives organization action more effectively than the former.

Section 5: Organization and Personnel

This section penetrates from "department" to "individual": employee Token details (name/employee ID/team/Token/number of events/used AI Agents and models), AI agent details (Tokens and number of employees aggregated by agent_type), and model details (aggregated by model).

"Exactly who is using it, what combinations they use, and how deeply they use it" form the factual basis for team owners to conduct one-on-one coaching and tool recommendations. The AI Agent/model details aggregate individual behaviors into tool profiles. This feeds back into tool selection and the accumulation of best practices.

Because you want to view "which agents/models a person has used", you need to aggregate multiple values into readable fields. The common syntax is to simply use array_agg(DISTINCT …). However, to ensure that the weekly report comparison does not generate noise, you must add another layer of array_sort to guarantee the stability of the outputs. Without this step, the exported CSV file will generate a large amount of noise between two differences because of the random jitter in the order of array elements.

SELECT
  d.show_name AS "Name",
  d.work_no AS "Employee ID",
  d.dept_name_2 AS "Team",
  sum(a.total_tokens) AS "Total tokens",
  sum(a.events) AS "Number of events",
  round(1.0 × sum(a.total_tokens) / nullif(sum(a.events), 0), 2) AS "Average tokens per event",
  array_join(array_sort(array_agg(DISTINCT a.agent_type)), ', ') AS "AI Agent",
  array_join(array_sort(array_agg(DISTINCT a.model)), ', ') AS "Model"
FROM dept_user d
JOIN active_user a ON d.work_no = a.user_id
GROUP BY d.show_name, d.work_no, d.dept_name_2
ORDER BY "Total tokens" DESC

Section 6: Skill & Tools

The three Top 10 vertical bar charts for the number of Skill calls, the number of Skill users, and the number of Tool calls constitute the most distinctive segment in the entire dashboard. The number of calls indicates the intensity with which a single Skill is used, and the number of users indicates its spread breadth within the team. A Skill with both high numbers is a methodology that is truly accumulated as an organization asset. A Skill with a high number of calls but a low number of users remains in the individual exploration stage.

However, to retrieve these rankings, you must first extract the Skill name from the file path. This itself is the most core SQL logic in this section. SKILL.md has many variations under the history folder structure (skills/<name>/SKILL.md, skills/<name>/v1.2/SKILL.md, and .qoderwork/<name>.skill.md). The fetch rules are as follows:

SELECT skill_name AS "Skill", count(1) AS "Number of calls"
FROM (
  SELECT
    CASE
      -- If the candidate is a version number, fetch another segment of the folder name upwards
      WHEN regexp_like(regexp_extract(path, '/([^/]+)/[^/]+$', 1), '^v?[0-9]+\.[0-9]+')
      THEN regexp_extract(path, '/([^/]+)/v?[0-9][^/]*/[^/]+$', 1)
      -- The candidate is the container folder name → fetch from the file name, and remove the suffix
      WHEN lower(regexp_extract(path, '/([^/]+)/[^/]+$', 1))
           IN ('skills','skill','.claude','agents','resources','.qoderwork','docs')
      THEN regexp_replace(regexp_extract(path, '/([^/]+)$', 1), '(?i)(\.skill)?\.md$', '')
      -- Default: take the folder name of the second to last segment
      ELSE regexp_extract(path, '/([^/]+)/[^/]+$', 1)
    END AS skill_name
  FROM <event table> t
  JOIN dept_user d ON t."user.id" = d.work_no
  WHERE regexp_like(t."gen_ai.tool.call.arguments.file_path",
'(?i)(/SKILL\.md|/[^/]+\.skill\.md)$')
)
WHERE skill_name IS NOT NULL AND skill_name <> '' AND upper(skill_name) <> 'SKILL'
GROUP BY skill_name
ORDER BY "number of calls" DESC LIMIT 1

Bypass the entire active_user, and directly JOIN the original event table, because gen_ai.tool.call.arguments.file_path is not in the CTE.

Section 7: Code Repository

If Skills answer "whether the methodology is reused", the repository answers "which code assets the AI-assisted coding investment actually flows into". The Repo Token Top10 horizontal bar chart and the Git Domain Token proportion pie chart provide two dimensions: whether tokens flow centrally into core business repositories or are scattered across experimental projects, and whether they are mainly consumed in internal GitLab (controlled assets) or external GitHub (code outside the border).

These two distributions directly serve the judgment of "whether the AI investment is aligned with the organizational strategy".

To support this perspective, an architectural trade-off must be resolved: git.repo and git.domain cannot enter the active_user CTE. If you add the repository to GROUP BY, the number of CTE rows expands from "person × day" to "person × day × repository", which instead slows down all downstream charts. Therefore, we revert to the original event table here to aggregate separately:

SELECT
  coalesce(nullif(t."git.repo", ''), 'unknown') AS "repository",
  sum(coalesce(t."gen_ai.usage.total_tokens", 0)) AS "Total Tokens"
FROM <event table> t
JOIN dept_user d ON t."user.id" = d.work_no
WHERE t."git.repo" IS NOT NULL AND t."git.repo" != ''
GROUP BY "repository"
ORDER BY "Total Tokens" DESC LIMIT 10

This is an example of "which dimensions should enter the CTE and which should be left out": a general pre-aggregation is not necessarily better just because it is wider.

Section 8: Token Concentration

Concentration is the most direct validation for all previous "total amount" metrics. When the "Tokens per capita" in Section 1 looks impressive, you must ask whether it is a benefit for all employees or a mean supported by a few top individuals. If the top 10% of individuals account for more than 70% of the tokens, the organization remains at the stage of "individual productivity of a few people" even if the total amount is large.

Numeric cards (top 10% or top 20% token proportion), audience layered vertical bar charts (three buckets: top 10%, 10% to 20%, and the bottom 80%), and daily concentration trend lines combine to answer "whether the concentration is improving or deteriorating." A continuous decrease in concentration means that AI coding is diverging from an individual capability to a common capability of the organization.

The key to making this judgment is to use a window function for ranking and then create buckets by percentile:

user_token AS (
  SELECT a.user_id, sum(a.total_tokens) AS total_tokens
  FROM dept_user d
  JOIN active_user a ON d.work_no = a.user_id
  GROUP BY a.user_id
  HAVING total_tokens > 0
),
ranked AS (
  SELECT user_id, total_tokens,
    row_number() OVER (ORDER BY total_tokens DESC) AS rn,
    count(*) OVER () AS user_count,
    sum(total_tokens) OVER () AS all_tokens
  FROM user_token
)
SELECT
  round(100.0 × sum(CASE WHEN rn <= cast(ceil(user_count × 0.1) AS bigint)
                         THEN total_tokens ELSE 0 END)
              / max(all_tokens), 2) AS "Top 10% Token Proportion"
FROM ranked

The daily trend version only has one additional PARTITION BY t, which changes "global ranking" to "ranking per day." cast(ceil(user_count × 0.1) AS bigint) ensures that the top 10% border can be correctly retrieved even if the total number of people is not divisible.

Conclusion

The answer provided in this topic is a three-layer progressive path:

Layer 1: unified semantics collection. The LoongSuite GenAI semantics specification solves the problem of "what standard to use for records," and LoongSuite-Pilot solves the problem of "how to collect from heterogeneous agents." Regardless of whether you use Claude Code, Copilot, or Cursor, events are written to SLS based on the same field structure, making the events comparable across tools and traceable across time.

Layer 2: Flexible analysis. The SLS dashboard uses SQL as the unique definition language. The calculation methods are completely self-controlled and can be queried in seconds. Public CTEs serve as the engineering foundation. This guarantees the calculation method consistency and maintainability of 30+ charts. From overview to structure, from trend to personnel, and from skill reuse to code repository and token concentration degree, the eight sections progress according to the analysis layers. Each layer provides context for judgments of finer granularity.

Layer 3: Actionable organization. The personnel dimension table exposes the empty areas of "registered but not reporting" through a LEFT JOIN. The concentration degree dashboard validates whether the per capita metrics are supported by top individuals. These are not just good-looking numbers. They are signals that can be directly converted into actions.

AI is an amplifier. The value of the measure layer is to let the organization clearly see what it is amplifying, and then decide where to go next.

Kernel-Level X-Ray Vision: CloudMonitor 2.0 Achieves Full-Stack Observability with Zero Code Changes

ObservabilityGuy — Tue, 11 Aug 2026 06:00:08 +0000

This article introduces how CloudMonitor 2.0 leverages OpenTelemetry eBPF Instrumentation (OBI) to achieve full-stack observability with zero code changes.

Background

In the cloud-native and microservices models, a production system often spans multiple language runtimes, such as Go, Java, Python, and Node.js. The deployment forms are scattered among containers, Kubernetes, and Serverless. To establish unified observability in such a heterogeneous environment, the traditional approach is to mount an intrusive Agent or SDK for each language. This involves modifying code, installing packages, aligning versions, and republishing. Each time a new service is connected, an engineering project is required. Under the fast research and development iteration pace, the cost of "connection means modification" is becoming increasingly unbearable.

At the same time, AI Agent applications are evolving from single Large Language Model (LLM) invocations to complex workflows with multi-step orchestration. A user request may trigger dozens of LLM invocations, Tool Calls, and AISearch. The invocation chain spans the Agent orchestration layer, LLM Provider, vector database, and external tools. Traditional application performance management (APM) is difficult to completely cover this. Zero-code observability solutions are equally indispensable in AI scenarios.

eBPF provides another idea. You can mount sandboxed probes in the Linux kernel to observe the network traffic, library function calling, and even system calls entering and leaving each process without modifying applications or restarting processes. Based on this capability, zero-code, cross-language, and low-overhead observability solutions are becoming a reality. OpenTelemetry eBPF Instrumentation (OBI) is the official answer provided by the OpenTelemetry community.

As an open source project officially maintained by OpenTelemetry, what OBI does can be summarized in one sentence. It utilizes the eBPF technology of the Linux kernel to automatically intercept and analyze network traffic and GPU operations entering and leaving the application without modifying any application code, and generates traces and metrics that comply with OpenTelemetry standards.

You can imagine it as a "X-ray glass" installed in the operating system kernel. Whether your application is written in Go, Java, Python, Node.js, or .NET, and regardless of what HTTP framework you use, what database you connect to, or which Large Language Model (LLM) you invoke, OBI can intercept communication at the kernel and library function layers, parse protocol semantics, and then output standard telemetry data. In terms of AI observability, OBI has built-in protocol-level tracking for four major GenAI Providers: OpenAI, Anthropic, Google Gemini, and Qwen. It can automatically detect LLM invocations and fetch Tool Call information from the response. It also supports the tracking of Rerank and AISearch operations, covering the core stages of the retrieval-augmented generation (RAG) pipeline.

It supports Linux amd64/arm64 architectures. The kernel requirement is 5.8+ (can be downgraded to 4.18+ for the RHEL series). The Deployment Method is flexible. You can run it as an independent process, deploy it as a Docker container, or deploy it as a Kubernetes DaemonSet.

Three Pillars: Application Monitoring, Network Monitoring, and Log Enhancement

The core capabilities of OBI revolve around three observability pillars.

Application Observability: Distributed tracing (Traces) + RED Metrics. It covers more than 15 mainstream protocols and scenarios, such as Web, databases, message queues, GenAI, and GPU. It automatically injects trace_id and span_id into JSON logs to achieve Trace-Log association.
Network Observability: L3/L4 network traffic monitoring and TCP/UDP traffic statistics. It supports GeoIP, reverse DNS, CIDR annotation, TCP RTT measurement, TCP connection failure statistics, and edge zone-level global metrics.
Log Enhancement: Transparently inject trace_id and span_id into JSON logs in a language-independent manner to achieve Trace-Log association (for more information, see the "Log Enhancement" section below).

Protocol Panorama: From HTTP to CUDA, Catching Everything in One Net

The core competitiveness of OBI lies in protocol-aware probing. It not only records that "there is a network request", but also deeply understands the semantics of each request. The following is the complete protocol matrix currently supported.

Web and RPC

GraphQL Yes — operation name and Type fetch

Cross-process context propagation for non-Go languages is uniformly implemented through the HPACK injection of the kernel mode tpinjector. For more information, see the "Cross-process Propagation" section.

Database

Message Queue

How Protocols Are Detected
How does OBI determine whether a segment of Transmission Control Protocol (TCP) payload is MySQL or Redis without decryption or relying on port conventions? The core lies in ReadTCPRequestIntoSpan (pkg/ebpf/common/tcp_detect_transform.go), which is a three-level waterfall match. It attempts to match in the order of "certainty from high to low" sequentially, and returns immediately upon a hit:

Kernel annotated (fastest): dispatchKernelAssignedProtocol, where the user mode directly switch event.ProtocolType. The kernel constants (common.go) are: MySQL=1, Postgres=2, Kafka=4, MQTT=5, MSSQL=6, NATS=7, AMQP=8. The SQL branch uses sentinel faults errFallback (fall back to the next level) or errIgnore (discard) for fine-grained control.
Deterministic generic match: detectGenericProtocol, which sequentially tries matchSQL → matchFastCGI → matchMongo → matchCouchbase → matchMemcached. SQL will first try the Request buffer, then the response buffer (when the response is hit, it invokes reverseTCPEvent to correct the direction).
Heuristic fallback (most prone to misjudgment, placed last): detectHeuristicProtocol, which tries matchRedis → matchMemcached → matchHTTP2 → matchNATS → matchAMQP → matchMQTT → matchKafkaFallback. The order itself is the accumulation of bug experience. For example, Hypertext Transfer Protocol 2 (HTTP/2) must be placed before Message Queuing Telemetry Transport (MQTT) because the heuristic of MQTT will falsely hit the connection preface of HTTP/2.

Several details to prevent misjudgment that are worth following:

SQL: First, it filters by printable American Standard Code for Information Interchange (ASCII) prefix (threshold = len("SELECT 1")), then searches for the keyword case-insensitively, and finally uses sqlprune.SQLParseOperationAndTable to fetch the operation and Table name. The validSQL requires "having an operation + (explicit Database (DB) Type or having a Table name)" to be counted.
Postgres: It validates the 5-byte message header, Type byte ∈ {Q,B,C}, and big-endian length in 0..3000. It also maintains the Least Recently Used (LRU) cache of the prepared statement and portal to revert the parameterized query.
HTTP/2 vs gRPC: First, isLikelyHTTP2 performs Request for Comments (RFC) 7540 frame-by-frame rationality validation (the upper limit of frame length is set to 1<<22, approximately 4 MB, as the validation tolerance value. Note that the RFC 7540 default frame size is 2^14, which is 16 KB. It uses the flags mask). Then, it differentiates them by looking at the content-type: application/grpc or grpc-status header.

Each protocol has a corresponding TCPToToSpan constructor to form a request.Span.

Language depth Integration: Not limited to the Network Layer

The detection of OBI is divided into two layers. The first layer is language-independent network-level tracing—applications of any language can obtain basic traces and metrics through TCP traffic blocking. The second layer is runtime-specific depth Integration—for specific languages and frames, OBI directly hooks library functions through uprobe to achieve more precise context propagation and trace association.

Go does not have ThreadLocal. How does OBI string together an invoke?

The goroutine of Go drifts among operating system (OS) threads. The thread local storage of traditional application performance management (APM) completely fails. The solution of OBI is to rebuild the parent-child relationship of goroutines in the kernel (bpf/gotracer/go_runtime.c + go_common.h). The solution hooks runtime.newproc1 to record which goroutine creates which goroutine and writes the record into the least recently used (LRU) ongoing_goroutines. When an outbound invocation needs to find the inbound request to which the outbound invocation belongs, find_parent_goroutine traces back up to 6 layers upwards along the parent chain. The deep nest is to be compatible with Kafka clients such as franz-go. Then, the solution hooks runtime.casgstatus to trail the status switch and attach the OBI context to the goroutine. This allows the kernel probe (kprobe) on the same OS thread to associate correctly.

The asyncio of Python uses single-threaded multiplexing. How does OBI distinguish concurrent requests?

The asyncio event loop of Python alternately executes hundreds or thousands of coroutines (Tasks) on the same OS thread. The traditional assumption of "one thread corresponds to one request" completely fails. What is more troublesome is that asyncio.to_thread() delivers the work to the thread pool. These worker threads do not have the asyncio.Task identity at all. The solution of OBI is to track the Task and Context objects of CPython in the kernel and rebuild the parent-child relationship of the coroutines.

The core consists of four groups of user-space probes (uprobes) (bpf/generictracer/python.c). The task_step tracks which Task the event loop switches to. The _asyncio_Task___init__ records the parent-child relationship and inherits the request connection when the Task is created. The PyContext_CopyCurrent attaches the replica to the corresponding Task when the Context is copied. Both create_task and to_thread trigger this action. The context_run recovers the Task identity when the worker thread activates the Context. Three Berkeley Packet Filter (BPF) Maps (python_thread_state / python_task_state / python_context_task) work together to cover four concurrent modes: await, create_task, gather, and to_thread.

The parent chain traceback mechanism (find_python_owning_server_trace) is similar to Go. The mechanism traces upwards along the parent pointer from the current Task for up to 4 layers until the mechanism finds the ancestor Task that holds the inbound request connection (server_traces_aux). Then, the mechanism can associate with the correct server span. The problem of Task address reuse is solved by the version counter. Each time the Task is initialized, the version auto increments. The version is snapshotted when the Context is attached. If the comparison is inconsistent during the search, the Task is determined to expire.

The entire solution is anchored on the CPython _asyncioand libpython symbols, and does not depend on the internal implementation of uvloop. The uvloop only replaces the I/O driver of the event loop. The semantics of asyncio.Task and contextvars remains unchanged. Therefore, the same group of probes is effective for both asyncio and uvloop, and no additional adaptation is required.

Cross-process propagation is uniformly completed in the kernel mode for non-Go languages.

The language runtime table in the previous text easily gives the impression that the cross-process context propagation is implemented by the language runtime of each language respectively. A more accurate expression is that the in-process context propagation is indeed exclusive to each language (Go goroutine traceback, Node async_hooks, Python asyncio, Ruby Puma queue, and Java/.NET tracking via OpenSSL/Java Virtual Machine (JVM) uprobe). However, the cross-process traceparent propagation is uniformly completed in the kernel mode by tpinjector for all non-Go languages (pkg/internal/ebpf/tpinjector + bpf/tpinjector/*.c). There are three methods:

HTTP/1 header injection: The sk_msg program rewrites the payload via the tail call chain and inserts the Traceparent: header. The SSL socket is directly skipped (the ciphertext cannot be modified).
HTTP/2 HPACK injection: Inject the HPACK-encoded traceparent per stream, and use the huffman fingerprint 0x3fa9851d6b21834d to detect existing headers.
TCP Option propagation (Custom TCP options): The options kind is 25. For outbound traffic, bpf_store_hdr_opt writes trace_id/span_id in the WRITE_HDR_OPT callback. For inbound traffic, bpf_load_hdr_opt retrieves the trace_id/span_id in PARSE_ALL_HDR_OPT, and writes the trace_id/span_id into incoming_trace_map (using the normalization connection_info_t as the key) for Consumption by the server-side server_trace_parent before the trace_id/span_id is deleted. During startup, sock_iter.c also fills established persistent connections into sockhash, so that old connections can also be injected. Deployment notes: TCP Option kind=25 belongs to an unassigned IANA number. Some firewalls, Server Load Balancers, and cloud platform middleboxes may strip unknown TCP options, which causes the propagation to fail in silence. We recommend that you verify the pass-through capability of TCP options in the target network environment, or prioritize the use of the HTTP header injection method (OTEL_EBPF_BPF_CONTEXT_PROPAGATION=headers).

This is controlled by OTEL_EBPF_BPF_CONTEXT_PROPAGATION: headers/tcp/all/disabled. finder.go determines whether to load the injector based on this configuration.

The Complete Lifecycle of a Span (Data Pipeline and Architecture)

How does a byte stream captured in the kernel become that trace in CloudMonitor 2.0? This is exactly the most hardcore engineering part of OBI. The user mode of OBI is not a large loop. Instead, the user mode is an explicitly declared, phased, and pluggable directed graph (DAG).

Top-level Skeleton: Three Independent Agents + errgroup

The entry point RunWithContextInfo (pkg/instrumenter/instrumenter.go) splits the three main pillars into three independent goroutines based on Feature Flags, and uses errgroup to attach them. If any goroutine crashes, the other two goroutines undergo a graceful shutdown when the context is canceled. The application observability pipeline is divided into three steps (pkg/internal/appolly/appolly.go): FindAndInstrument (Search and attach probes) → ReadAndForward (Start the processing pipeline) → WaitUntilFinished.

Pipeline Orchestration Framework: swarm (two-stage Start)

OBI has developed a minimalist node orchestration framework pkg/pipe/swarm. The core of the framework is the two-stage semantics of "instantiate all first, and then Run uniformly".

In the first stage, Instancer.Instance(ctx) invokes the InstanceFunc of each node in sequence. If an initialization failed error occurs, the framework immediately cancels the process and returns an error as a whole. No RunFunc will Start, which avoids the incomplete Status of a "half-Start". In the second stage, Runner.Start(ctx) spawns a goroutine for each node, which can be configured with WithCancelTimeout. After the context is canceled, if a node fails to exit within the timeout, Done() returns CancelTimeoutError and identifies the specific zombie node.

Inter-node communication: msg.Queue (fan-out queue with deadlock detection)

Nodes do not invoke each other directly. Instead, nodes pass messages through the generic queue msg.Queue[T] (pkg/pipe/msg/queue.go):

fan-out: A queue can be Subscribed by multiple downstream nodes. SendCtx delivers the same message to all subscribers. When no subscriber exists, the message is directly discarded and does not block the sender.
Bypass (zero-cost short circuit): When a branch is configured for shutdown, input.Bypass(output) directly transfers the upstream subscribers to the downstream nodes. The disabled node does not perform a dry-run, but physically disappears from the graph.
Deadlock self-check: SendCtx has a built-in sendTimeout timer (default 1 minute). An alerting is triggered if a channel write block of a subscriber times out. In PanicOnSendTimeout pattern, it directly panics and prints the A->B->C path.
Multiple producers shutdown: ClosingAttempts(n) + MarkCloseable() reference counting. It truly closes only after all producers are marked as closeable.

Complete Directed Acyclic Graph (DAG) for Application Observability

The Build() function in pkg/internal/appolly/instrumenter.go explicitly pieces together the entire graph:

[per-process eBPF tracers]
   | (processes share the Ring Buffer)
   v
   ringBufForwarder  (reader goroutine + parser goroutine, object pool 2 × BatchLength)
   | tracesInput  (batch=100 / 1 s / 3 s idle-flush)
   v
ReadFromChannel -> Routes -> KubeDecorator -> DockerDecorator -> NameResolution -> AttributesFilter
   |
   v  exportableSpans  ===== fan-out =====
   |-- OTEL Traces Exporter
   |-- Printer (debug)
   |-- SpanNameLimiter -> [OTEL Metrics | SvcGraph Metrics | Prometheus]
   `-- BPF Metrics

Two engineering design points: (1) The Metric sub-pipeline is started on demand. The setupMetricsSubPipeline is executed only if a Metric export is actually configured. (2) Special timeout for K8s decorators. The routerToKubeDecorator queue takes max(InformersSyncTimeout, ChannelSendTimeout) because the full informer snapshot must be pulled first during startup, and it cannot be mistakenly killed by the default deadlock detection.

Transport from kernel to user mode: ringbuf forwarder with dual goroutines and an object pool.

This is the ingest endpoint of the entire pipeline, and also the most Performance-sensitive place (pkg/ebpf/common/ringbuf.go, generic ringBufForwarder[T], where T=Span on the application side and T=Record on the Stats side reuse the same Code):

Read/parse separated producer-consumer: The readerLoop is only responsible for ReadInto the original record, and the parserLoop is responsible for parsing it into a Span. They pass the slot indices through two channels: freeIdx and workIdx.
Object pool avoids Garbage Collection jitter: Pre-assign poolSize = 2 * BatchLength records for reuse (one batch is for the parser, and one batch is for concurrent padding by the reader).
Batch commit + timeout fallback: It flushes when BatchLength (default 100) is reached. If it is not reached, the BatchTimeout (default 1 s) ticker acts as a fallback. In addition, flushOnAvailableBytes actively flushes by checking the residual bytes of the ringbuf every 3 s to prevent Data from being stuck in the kernel under low traffic.
Shared ringbuf: Hundreds or thousands of probed processes share a SharedRingBuffer, and thousands of closers execute concurrent Close() upon exit.

When any internal queue is blocked, OBI proactively prompts the full set of tuning knobs: OTEL_EBPF_OTLP_TRACES_BATCH_MAX_SIZE, OTEL_EBPF_OTLP_TRACES_QUEUE_SIZE, OTEL_EBPF_CHANNEL_BUFFER_LEN, OTEL_EBPF_CHANNEL_SEND_TIMEOUT, OTEL_EBPF_BPF_BATCH_LENGTH, and OTEL_EBPF_BPF_BATCH_TIMEOUT.

One-sentence summary: OBI = a swarm-orchestrated DAG + a set of fan-out queues with deadlock Detection + a dual-goroutine object pool ringbuf forwarder. The Start is either fully Succeeded or fully rolled back. The shutdown can locate zombie nodes. The disabled features physically disappear from the graph.

GPU/CUDA Tracking: Covering Kernel Launches and VARM Operations

In addition to depth probing at the network protocol layer, OBI extends observability to the GPU Compute realm. By hooking libcuda.so via uprobes, it can track the core operations of NVIDIA CUDA: kernel launches, graph launches, memory assignments, and memory copies. This is highly valuable for GPU clusters running AI training and infer Jobs.

The configuration of GPU tracking is very simple: OTEL_EBPF_CUDA_MODE=auto automatically detects whether the System has CUDA libraries and enables tracking.

GPU tracking uses uprobes to hook libcuda.so combined with an independent ring buffer (bpf/gpuevent on the kernel side, and pkg/internal/ebpf/gpuevent in user mode). An engineering detail worth pointing out is that CUDA events enter the processing pipeline through the same generic forwarder ringBufForwarder[T] (pkg/ebpf/common/ringbuf.go) as Network Spans. Therefore, CUDA events naturally share the same batch commit, backpressure control, and graceful shutdown mechanisms, eliminating the need to maintain separate transport logic for the GPU. This also explains why OTEL_EBPF_CUDA_MODE=auto can achieve zero-extra-configuration integration: it essentially just attaches an additional data source to the existing processing DAG.

Network Layer Observation: Traffic, Association, and Quality

In addition to application-level distributed tracking, OBI also provides Network-level observability.

Network Traffic Monitor (NetO11y)

Based on TC hooks, OBI catches L3/L4 network packets, parses IPv4/IPv6 and TCP/UDP headers, and generates network traffic Metrics. The Data pipeline supports rich decoration capabilities: Kubernetes metadata, reverse DNS, GeoIP geolocation, and Custom CIDR range annotation. This is suitable for intra-cluster and inter-cluster traffic Analysis, network topology visualization, and security audits.

Typical Troubleshooting Scenarios

Suspicious Outbound Connections Discovery (security audit): An unknown IP address suddenly appears in the service topology with a large amount of data transmission (such as 2.3 GB/h) to the database. By combining rDNS reverse lookup and GeoIP geolocation, you discover that the Target is overseas. This immediately triggers a security alert.
Cross-zone traffic spike troubleshooting: Cross-zone traffic suddenly doubles during a certain period. After aggregating by CIDR annotation, you discover that a routing configuration Change for a certain service causes all Requests to detour to another AZ. After the issue is fixed, bandwidth costs drop back to normal.
service Dependency discovery: Without any configuration, OBI automatically generates a cluster-wide inter-service traffic topology. When a service is abnormal, you can immediately see all of its upstream and downstream Network relationships and traffic changes.

Statistical Metrics (StatsO11y)

By using kprobes or tracepoints, OBI collects node-level TCP RTT (obi_stat_tcp_rtt_seconds) and the Count of failed TCP connections (obi_stat_tcp_failed_connections) to help you monitor the underlying network quality.

Typical Troubleshooting Scenarios

Fast network congestion delimitation: The response of a service slows down, and the Transmission Control Protocol (TCP) round-trip time (RTT) surges from 2 ms to 180 ms. You do not need to troubleshoot layer by layer. You can directly view the RTT Metric and perform aggregation by Target IP address. Within 1 minute, you can localize the congestion to the vSwitch where the payment-gateway edge zone is located.
vSwitch/CIDR block fault localization: The number of failed TCP connections suddenly erupts in the 10.0.2.0/24 CIDR block, while other CIDR blocks are Normal. You can immediately infer that the vSwitch or Ingress of this CIDR block has issues, and notify the Network team to intervene.

Log Enrichment: Kernel Block + Transparent Injection of trace_id

The log enrichment of OBI is a language-independent capability. Regardless of what language or log frame the application uses, as long as it writes logs in JavaScript Object Notation (JSON) format to stdout/stderr or pipe, OBI can block them at the kernel layer and transparently inject trace_id and span_id. With zero modifications to the application Code, the Trace foreign key field automatically appears in the log file.

Implementation principle (bpf/logenricher/logenricher.c): Hook the tty_write (terminal Outputs) and pipe_write (pipe Outputs, covering the docker logs/kubectl logs scenarios) kernel functions via kprobe. When the write operation of the monitored process is triggered, the Berkeley Packet Filter (BPF) program reads the raw log Content (maximum 8 KB) from the user mode buffer. At the same time, it retrieves the trace context of the current thread or coroutine via obi_ctx, then uses bpf_probe_write_user to erase the raw log (padding with zero bytes), and sends the event via ringbuf to the user mode.

Known limitations: bpf_probe_write_user directly modifies the user mode memory, and there are two threats to follow: (1) A time window exists between the kernel zero-erasing and the user mode write-back. If the process crashes or the log collector happens to read the Outputs during this period, log loss or blank lines may occur. (2) Some security hardening kernels (such as those with the Lockdown pattern enabled) disable this function. Before deployment, you must confirm that the kernel configuration allows this operation.

After the user mode processor (pkg/internal/ebpf/logenricher) accepts the event: It attempts to parse the log line into JSON. If it is Succeeded, it injects the trace_id and span_id fields into the object (if they already exist, it does not overwrite them), and after it serializes them, it writes them back to the raw Outputs path of the application (TTY pts or pipe fd). If it is not JSON, it writes the log line back as is. The write-back path is parsed via path_resolver from the kernel file->f_path (TTY scenario) or located from /proc//fd/ (pipe scenario), and a Least Recently Used (LRU) cache is used to avoid repeated open operations.

Several engineering highlights: On the kernel side, the current fd is pre-recorded via ksys_write/do_writev kprobe, and when pipe_write is triggered, the file descriptor (fd) can be associated with the correct pipe. The asynchronous writer (ShardedQueue) performs shards by File Path, and the writing of the same file is Strict serial to guarantee the log line sequence. If the application already comes with the OpenTelemetry (OTel) software development kit (SDK) to export Traces (ExportsOTelTraces), only trace_id is injected, but not span_id, to avoid conflicts with the span of the application itself.

Use OBI (OpenTelemetry Non-intrusive Monitoring) in CloudMonitor 2.0

One-click Integration

Via the Integration Center of CloudMonitor 2.0, select OpenTelemetry Non-intrusive Monitoring, and select one-click integration for the cluster [1].

Application Details
After integration, you can view the number of Requests, Number Of Error, and Duration of the application, as well as the invoke details of the API.

Trace Analysis
In Trace Analysis, you can view the complete Trace link, making issues clear at a glance.

Network Monitoring
Taking the "source service → Target service" link as the core perspective, it displays the network traffic rate and TCP round-trip time (P50/P95/P99) between microservices in the Kubernetes (K8s) cluster in real-time. It is used for network quality inspection, delay abnormal localization, and traffic Distribution Analysis.

To better use the capabilities of OBI, you can achieve one-click integration via the Integration Center of CloudMonitor 2.0. Next, we will gradually supplement Artificial Intelligence (AI) Agent observability capabilities [2] and more Network Monitoring capabilities into OBI. You are welcome to build together.

[1] CloudMonitor 2.0 Integration Center: https://www.alibabacloud.com/help/cms/cloudmonitor-2-0/kan

[2] https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/issues/1854

Electron Monitoring: Making Desktop Agent Monitoring Accessible

ObservabilityGuy — Tue, 11 Aug 2026 05:17:38 +0000

This article explains how to use an SDK to achieve full-link observability for Electron applications.

Introduction: The "Monitoring Blind Spot" on the Desktop Side

Your Electron application is published. Users are using the application, business is running, and everything looks good—until one day, customer service forwards a user feedback: "The application crashed suddenly, and nothing was saved."

You open the log and find that the main process is in silence. The window.onerror of the rendering process is configured, but those data are scattered in the network requests of various windows, and a complete fault link cannot be pieced together at all. What is more tricky is that the .dmp file left by the native crash requires a dedicated Symbol Table service to parse, and you have not even figured out where to transmit the crash data.

This is not an isolated case. Desktop applications with tens of millions of daily active users, such as Visual Studio Code (VS Code), 1Code, OpenCode, and Cherry Studio, are all built on Electron. However, when we talk about observability, our focus is often on the server-side and the browser side. The desktop side is almost a monitoring blind spot.

This article aims to solve this problem: how to use a set of SDK to transform an Electron application from a "black box" to "full-link observability", and make desktop Agent monitoring truly accessible.

Why is Electron Monitoring Difficult?

Before discussing the solution, you can first understand the essence of the problem. The monitoring challenge of Electron is rooted in its unique dual-process architecture:

The first layer of difficulty: the main process and the rendering process are two worlds. The main process runs on Node.js, and is responsible for native Application Programming Interface (API), system interaction, and process management. The rendering process runs on Chromium, and carries User Interface (UI) and business logic. The two are bridged through inter-process communication (IPC). However, from a monitoring perspective, they are two independent runtime environments—the exception types are different, the performance metrics are different, and the data collection methods are also different. The traditional frontend Real User Monitoring (RUM) SDK can only cover the rendering process, and the monitoring of the main process is completely missing.

The second layer of difficulty: a native crash is a "black box". An Electron application may experience a native crash because of V8 engine bugs, Native module abnormalities, or system resource exhaustion, leaving a binary .dmp dump file. Parsing it requires a minidump Resolver and a Symbol Table service, which is a completely unfamiliar realm for most frontend teams.

The third layer of difficulty: the data reporting path is unreliable. The network requests of the rendering process may break because of page destruction or window shutdown. If each window reports data independently, it not only wastes resources, but also easily loses key events—especially at the moment when a crash occurs.

The fourth layer of difficulty: IPC communication lacks observability. More and more Electron applications adopt frameworks such as TypeScript Remote Procedure Call (tRPC) to implement type-safe communication between the main process and the rendering process. However, there are almost no off-the-shelf monitoring solutions for the performance, faults, and Tracing Analysis of such IPC invocations.

These pain points stack together, making Electron monitoring a realm where "everyone knows it should be done, but no one wants to be the first to step into the pit".

Breakthrough Idea: One `init()`, Full-end Observability

Our solution is @arms/rum-electron—the Electron SDK of Alibaba Cloud CloudMonitor. The core design concept can be summarized in one sentence: a single line of init() invocation in the main process automatically covers the comprehensive monitoring of the main process and all rendering processes.

SDK dual-process architecture: the main process acts as the data convergence center, rendering process events backflow through the IPC Bridge, and are uniformly reported by the main process

This architecture has three key design decisions:

The main process acts as the data convergence center. All events—whether from the main process itself or the rendering process—ultimately converge to the main process, and are uniformly reported by ElectronReporter. The Browser SDK of the rendering process does not initiate any network requests, and events backflow to the main process through the arms:rum-bridge IPC channel. This ensures that data is still not lost in extreme scenarios such as window shutdown or page crash.

Zero-configuration automatic injection for rendering processes. The SDK acts as a listener for the web-contents-created event of Electron, and automatically injects the Browser SDK script and IPC Bridge at the dom-ready timing of each BrowserWindow. Developers do not need to modify the preload script, import the SDK in the rendering process, or manually configure anything.

Three-stage Build and single-package distribution. The Preload script, Browser SDK, and WebAssembly (WASM) crash parse engine are all embedded into the main package during the Build. Users can directly use the package after they run npm install @arms/rum-electron, without additional file download or configuration steps.

Six Core Capabilities to Eliminate Monitoring Blind Spots One by One

Capability 1: Zero-Configuration Automatic Injection — Out-of-the-Box

This is probably the part that you care about the most. The integration Code is as follows:

// main/index.ts —— Top of the file
import armsRum from '@arms/rum-electron';

armsRum.init({
  endpoint: '<your-endpoint>',
});

There is no second step. After the main process invokes init(), all BrowserWindows automatically obtain complete rendering process monitoring capabilities — page views (PVs), Performance, Web Vitals, Blank Screen Detection, API Requests, long tasks, and User interactions, without missing any.

The SDK automatically adapts to the contextIsolation security policy. When the contextIsolation security policy is Enabled, the SDK uses contextBridge.exposeInMainWorld() to safely expose the Inter-Process Communication (IPC) channel. When the contextIsolation security policy is shutdown, the SDK directly mounts the IPC channel to the window object. In scenarios where a Custom partition is used, you can pass an additional partition field when you invoke init.

Capability 2: Local Crash Parse with Rust WASM Driver

When a native crash occurs in an Electron application, the System Generates a .dmp dump file. Traditional solutions (such as Sentry) upload this file to the server-side for symbol parse. Our approach is different — we moved the parse engine to the local client.

Specifically, we use the rust-minidump series of libraries based on Rust, compile them into WebAssembly by using wasm-pack, and directly complete crash stack trace and module parse locally after the application restarts. The WASM engine is embedded in the JS module with Base64 encoding (about 1.5 MB). You do not need additional file distribution or server-side symbol services, and the raw data of the crash does not leave the client.

The parse Result includes the following items: detection of the crash thread, the complete stack of all threads (module name, instruction address, offset, and frame reliability), the List of loaded modules (base address, size, and debug identifier), system info, and more. This means that your Security Compliance team can breathe a sigh of relief — sensitive crash stack Data is completely processed locally.

Capability 3: tRPC — IPC Monitoring Is No Longer a Blind Spot

More and more Electron applications use electron-trpc to implement type-safe communication between processes. However, at the monitoring layer, the Performance, faults, and Tracing Analysis of invokes of tRPC procedures have always been a blind spot.

@arms/rum-electron easily solves this problem by using instrumentTRPC():

import { initTRPC } from '@trpc/server';
import armsRum from '@arms/rum-electron';

// Wrap a layer around t, and all procedures automatically include monitoring
const t = armsRum.instrumentTRPC(initTRPC.create());

export const appRouter = t.router({
greeting: t.procedure.input(...).query(...),// automatic monitoring
chat: t.procedure.input(...).mutation(...),// automatic monitoring
});

The bottom layer uses JavaScript Proxy to block the property access of t.procedure and automatically injects the monitoring middleware at runtime. The procedure definition on the business side does not need to be modified at all, and existing chained middleware (such as auth) is not affected. The collected data snaps to the OpenTelemetry RPC semantics conventions (rpc.system = 'trpc') and seamlessly achieves filter interaction with the backend application performance management (APM).

This is currently the only Electron SDK on the market that natively supports monitoring on the tRPC server side.

Capability 4: Tracing Analysis

An Electron application is not an isolated island. It needs to invoke backend APIs, access AI inference services, and request third-party APIs. To connect the complete invocation chain from the client to the backend, distributed tracing is required.

The SDK supports five propagation protocols: W3C Trace Context, B3, B3 Multi, Jaeger, and SkyWalking. The fetch requests of the main process and the tRPC procedure share the same tracing decision. You can perform fine-grained control over the sampling policy by domain name:

armsRum.init({
  endpoint: '<your-endpoint>',
  tracing: {
    enable: true,
    sample: 10,                                   // global 10% sampling
    propagatorTypes: ['tracecontext', 'b3'],    // Optional: 'b3multi' | 'jaeger' | 'sw8'
    allowedUrls: [
      { match: /^https:\/\/api\.example\.com/, sample: 100 },  // core API 100%
      /^https:\/\/cdn\.example\.com/,                         // CDN 10%（use global）
    ],
  },
});

The tracing header is automatically injected into outbound requests, and the business code is completely unaware of it. This means that when user feedback indicates that "AI replies are too slow", you can trace all the way from the tRPC invocation of the Electron client to the Large Language Model (LLM) inference service of the backend, and accurately locate which step is slow.

The following is a complete chain from the frontend -> Agent -> MaaS.

You can complete the integration of Tracing Analysis for both the frontend and backend simultaneously.

Capability 5: Innovative Memory Aggregation Window - The Sentinel of Memory Leak

Electron desktop applications often run for a long time (such as IDEs, design tools, and IM clients), and a memory leak is the Sword of Damocles hanging over the head of every team.

We designed a memory monitoring solution of "high-frequency background sampling + low-frequency window Aggregation Reporting + event trigger fallback":

Every 10 seconds, you can perform synchronous sampling on

app.getAppMetrics()

and accumulate the data into the accumulator in the memory, with a CPU overhead < 0.1%.

Every 30 minutes, it Outputs two management events: memory_max (peak) and memory_avg (mean), with only about 5 management events per hour.

When a crash or exit occurs, the current memory state is immediately flushed. This ensures that out-of-memory Analysis is well-documented.

Cross-window mean trajectories track baseline drifts to detect memory leak Trends in advance.

The brilliance of this solution lies in the following aspect: it provides a memory profile with sufficient granularity at an extremely low overhead (5 management events per hour and < 0.1% Central Processing Unit (CPU)). When you detect that the memory_avg of a Version climbs window by window, it is highly likely a signal of a memory leak.

Capability 6: Comprehensive Abnormal Protection Network

At the main process level, the SDK Builds a three-layer abnormal protection network: uncaughtException catches unhandled exceptions, unhandledRejection blocks Promise rejections, and console.error performs non-intrusive blocking. The three-layer fallback ensures zero omissions of main process faults.

At the rendering process level, the automatically injected Browser SDK covers scenarios such as frontend abnormalities, unhandled Promise rejections, and Blank Screen Detection. All anomalous activities Backflow to the main process through the Inter-Process Communication (IPC) Bridge for unified Reporting. Even if the rendering process crashes, the previously collected management events will not be lost.

All function blocks adopt a non-intrusive monkey patch design, which can be completely reverted after the SDK is uninstalled. It comes quietly and leaves without a trace.

Compared with Competitors, What Are the Advantages?

Below, we will make an objective comparison between @arms/rum-electron, the Sentry solution that has the deepest investment in the Electron monitoring realm, and the Electron Support status of mainstream observability platforms in China:

Comparison 1: vs Sentry Electron SDK

Sentry is currently the vendor with the deepest investment and the most active community in the Electron monitoring realm. It has excellent capabilities such as the dedicated SDK @sentry/electron, an independent product page, session replay, and event loop block Detection. It has a very high degree of recognition in the Developer Group outside China. More than 75% of Electron developers have integrated Sentry in their production environments (official Data).

In terms of feature coverage, the differences between the two are mainly reflected in the following dimensions:

Simply put, Sentry has obvious advantages in the ecosystem outside China and community Size. However, in dimensions such as Data compliance, tRPC monitoring, crash Data Sovereignty, and Tracing Analysis protocol coverage,@arms/rum-electron provides a solution that is more suitable for domestic enterprises and teams with Strict requirements for data security.

Comparison 2: vs General Frontend Real User Monitoring (RUM) SDK

You may ask: Can I directly use a general frontend RUM SDK? It is technically possible, but the experience is completely different:

Simply put, a general frontend RUM SDK can see half of the Electron application (the rendering process), while @arms/rum-electron gives you a panoramic view.

Quick Intergation: Simpler than You Think

The following are the complete Integration steps:

Environment requirements: Electron >= 28 (The SDK uses the session.registerPreloadScript() API. Lower Versions automatically downgrade to session.setPreloads().)

Step 1: Install

npm install @arms/rum-electron

Step 2: Main process entry initialization

import armsRum from '@arms/rum-electron'; 
import { app, BrowserWindow } from 'electron';

armsRum.init({
  endpoint: '<your-endpoint>',  // Obtained from the  CloudMonitor console
  env: 'prod',
  version: '1.0.0',
  // Optional: enable Tracing Analysis (put the tracing configuration directly in the init parameter)
  // tracing: {
  //   enable: true,
  //   sample: 100,
  //   propagatorTypes: ['tracecontext'],
  // },
});

app.whenReady().then(() => {
  const win = new BrowserWindow({
    webPreferences: {
      // No SDK-related configuration is required
    }
  });
  win.loadURL('https://your-app.com');
});

Step 3 (optional): enable TypeScript Remote Procedure Call (tRPC) monitoring

const t = armsRum.instrumentTRPC(initTRPC.create());

It is that simple. You do not need to modify preload or import anything in the rendering process. Zero configuration, zero intrusion, and zero additional manual install.

Performance and Overhead: Lightweight Enough to Be Ignored

The value of the monitoring SDK is to help Search for problems, not to become a problem itself. The following are the resource overhead Data of @arms/rum-electron:

The SDK follows the "principle of minimum interference" everywhere in its design. The memory Collection uses an accumulator with O(1) assign. The WebAssembly (WASM) initialization is delayed until it is first used. The Remote Configuration adopts the launch-first pattern and does not block the Application Start. All function blocks Support the restore revert, which ensures that the existence of the SDK does not become a burden on the application.

Scenarios

The scenarios that @arms/rum-electron targets include but are not limited to the following:

Enterprise-level Electron desktop applications. These applications include collaboration tools such as Slack, Discord, and Notion, and various internal enterprise tools and low-Code platforms. You can uniformly monitor health, Performance, and stability, and use Remote Configuration for dynamic adjustment of the sample rate.

AI desktop applications. These applications include AI chat assistants, AI programming tools, and AI design tools. The tRPC monitoring covers the AI infer invoke trace. The memory level tracking helps Search for the memory leak caused by Large Language Model (LLM) load. The distributed tracing connects the client to the AI backend.

Desktop applications with a long lifecycle. These applications include integrated development environments (IDEs), design tools, and trading platforms. You can use an aggregation window of 30 minutes to track baseline drift, immediately flush the memory site when a crash occurs, and Search for a memory leak in advance.

Industries with Strict requirements for data security. These industries include finance, government, and healthcare. The local parse of the crash stack is performed by WASM, and the raw data does not leave the client. Alibaba Cloud Application Real-Time Monitoring Service (ARMS) has its deployment in China to meet graded protection requirements.

Conclusion

Electron gives Web Developers the ability to Build cross-platform desktop applications, but it also introduces unique monitoring challenges such as dual-process architecture, native crash, and inter-process communication (IPC). In the past, these challenges were either ignored or required piecing together multiple sets of tools to barely cover them.

The Target of @arms/rum-electron is simple: **Make the monitoring of Electron applications **as simple as that of Web applications. These capabilities, including the out-of-the-box integration experience, full coverage of the main process and rendering process, local parse of crash Data, native Support for tRPC, and five-protocol Tracing Analysis, are integrated into one Node Package Manager (npm) package, which is ready to use after you install it.

Desktop Agent monitoring is now within reach.

Experience now: Go to CloudMonitor for the Creation of a Real User Monitoring application, and you can start the integration after you retrieve the endpoint.

Technical documents: See the complete configuration reference document.

Community communication: You can join the DingTalk group to communicate with the Alibaba Cloud observability team about Electron monitoring practices.

Can the Log Be Changed? SLS Logstore Adds Native Support for Log Updates and Deletes

ObservabilityGuy — Mon, 10 Aug 2026 07:39:07 +0000

This article introduces native log update and delete capabilities in Alibaba Cloud SLS Logstore.

Introduction

We are the Alibaba Cloud SLS team. SLS is Alibaba Cloud’s one-stop log and observability platform, responsible for the ingestion, storage, querying, analysis, and shipping of massive volumes of enterprise log data every day. For the past decade, logs written to LogStore have been immutable. Recently, however, we introduced native update and delete capabilities for existing data. In this article, we would like to share the story behind this decision and the engineering trade-offs we navigated along the way.

I. Foreword

"Can logs be modified?"

Over the past decade, the answer from Simple Log Service (SLS) has been very clear: they cannot be modified, and they should not be modified.

Because the underlying layer of a Logstore is an append-only design: what is written becomes history, which is also the core advantage of a log system in terms of performance, stability, and audit credibility.

Then why do we add the capabilities of modification and deletion to it this time?

To answer this question, we first review the development history of logs over the years, and you can see that the uses of logs have been continuously expanded.

Phase 1: Logs Are for Viewing

The earliest logs are just for humans to view. If a service encounters a fault, you can log on to the machine to run tail -f, run grep for a few keywords, and see which line reports an error. In the generation of centralized log systems, the task of "viewing logs" has been moved to the browser, but the essence remains unchanged: the lifecycle of logs is still very simple, which includes writing, viewing, and then being cleaned up after the logs expire.

This usage mode brings a very clear design premise to the log system: append-only, no modification.

The write path does not require edit locks or version snaps. The storage layout is naturally organized by time. Downstream consumers only need to record the offset to recover the consumption progress. The high throughput, low cost, and stability of the log system are largely built on this set of rules.

Phase 2: Logs Are for Analysis

Later, logs begin to be structured, indexed, and queried and analyzed.

SLS also continuously complements capabilities such as SQL, Structured Process Language (SPL), aggregate functions, machine learning operators, alerting, dashboards, delivery, and consumption at this stage. Logs are no longer just texts that are opened only when you troubleshoot faults, but gradually become data sources in operational systems.

Risk control systems subscribe to behavioral logs for real-time rule matching. Monitoring systems compute P99 and QPS metrics based on gateway logs. Security teams perform association analysis in audit logs. BI teams feed transaction logs to reports after aggregation.

Logs have changed from being "read by humans" to being "read by programs", and continuously enter the links of analysis, alerting, reports, and automated consumption.

However, this stage still doesn't challenge the append-only foundation. The write path remains a simple append operation—what changes is the addition of stronger indexing, computation, and consumption capabilities layered on top. SLS's goal is to make queries faster, analytics more comprehensive, and subscriptions smoother—not to modify data that has already been written.

Phase 3: "Logs" Carry More Business Scenarios

As logs carry more and more business semantics, requirements such as correction, backfill, overwrite, and cleanup, which rarely appeared in log systems in the past, also begin to become common.

Typical scenarios include:

After the metering and billing details are generated by billing cycle, some records need to be corrected at the end of the month because of discounts, refunds, or account adjustments.
For user features or content tags produced by AI pipelines, historical data needs to be recalculated in batches and old results need to be overwritten after the model is upgraded.
After the risk control system publishes a new version of the scoring rule, it is found that the old rule evaluates the threat levels of some users too low, and batch correction is required.
Information such as user feedback, manual annotations, and quality scores in LLM or Agent applications usually arrives with a delay after the request ends, and needs to be backfilled to the original records.
After collection configurations are faulty or test traffic mistakenly enters the production environment, targeted cleanup is required based on conditions. In the past, when users encountered these requirements, they usually needed to handle them by using soft deletion and rewrites, supplementing variable fields in external KV or databases, or appending new records and then letting downstream components merge them on their own. The solution can work, but the link is longer, the maintenance cost is higher, and the consistency is more difficult to guarantee.

If a Logstore can natively support partial updates and deletes, this type of scenarios can be completed in a closed loop within the Logstore, without the need to detour to external systems.

II. Design Tradeoffs

If you look closely at the preceding scenarios, the data entity still has typical log features: it is continuously generated, has a large write volume, is naturally organized by time, and is usually semi-structured. At the same time, it requires queries, analysis, alerting, and reports, and also needs to be consumed and delivered by downstream components.

It is exactly these features that allow the Logstore to gradually form today's capability combination: elastic capacity, flexible schema, high-throughput append writes, low-cost storage, strong query and analysis, and a complete consumption and deliver ecosystem.

Now there are requirements for partial updates, but the core features remain unchanged: they are still mainly massive appends, supplemented by a small number of corrections.

Therefore, a Logstore natively supports update and delete operations. The core design principle is as follows:

Append writing remains the main path, and modification and deletion are supplementary paths for existing data.

Specifically, this is reflected in the following design decisions.

Modification Capability Needs to Be Explicitly Enabled

Only when you set enableModify=true in the Logstore properties, the modification and deletion capabilities are enabled (they cannot be disabled after they are enabled).

For data that does not need to be modified, such as access logs, audit logs, and trace details, you can continue to use the append-only model.

Raw Data Ingestion Requests Remain Append-Only

Even after enableModify=true is set, new data is still ingested via direct append writes. This means ingestion requests won't deduplicate based on a business primary key or automatically overwrite existing records. To modify or delete ingested data, you must explicitly issue update or delete requests using __rowid__ or specific query conditions.

This design ensures that the normal ingestion path still maintains high throughput, low cost, and stable consumption semantics.

Accordingly, real-time LogHub consumption and delivery jobs still operate on the raw ingestion stream—they have no awareness of any subsequent updates or deletions.

Modifications and Deletions Are Synchronously Visible

To ensure high performance for normal ingestions, indexes are built in near real-time. There may be a very short delay between log ingestion and queryability. Because normal ingestions follow append-only semantics, this does not affect the ingestion order.

The modification and deletion operations added this time take effect synchronously. That is, after the API returns a success, you can see the corresponding effects in subsequent query and analysis.

This is particularly important for continuous partial updates. After the previous update returns a success, the subsequent update can ensure that it continues to take effect based on the results of the previous update. This avoids the semantic ambiguity of whether the subsequent update sees the previous update.

Query Experience Remains Unchanged

After enableModify=true is set, query and analysis are still completed through Search, SQL, and SPL, and the usage remains unchanged.

The modified or deleted data will reflect the corresponding results in subsequent query and analysis. For businesses that use Logstores to retrieve data, analyze data, generate reports, and manage the backend, the original query links do not need to be refactored because of the update and delete capabilities.

III. Capability Forms

After enableModify=true is set, Logstores provide modification and deletion capabilities in two dimensions: operating on a single record at a fixed point based on __rowid__, and performing batch operations on a batch of records based on query conditions.

Rowid: Row-Level Addressing Identity

After enableModify=true is set, each log is assigned a stable internal __rowid__ identity rowid. The user obtains the __rowid__ through Search, SQL, or SPL queries, and then uses it to precisely specify which record to modify or delete.

Note that __rowid__ is an internal identity used for row addressing in Logstores. It is not a business primary key or a primary key in a database. It is only used to locate existing records, and cannot be used for "insert by ID" or "overwrite by ID".

Therefore, operations based on __rowid__ are more suitable for scenarios where you query first and then modify. For example, a batch of records is retrieved from the management backend, the user selects one of them, and then precisely writes it back based on __rowid__.

Operations Based on Query Conditions: Batch Correction Using Business Fields

If you want to operate based on business fields such as request_id, batch_id, order_id, andtrace_id, you can use the conditional update or delete method.

The caller only needs to specify the time range and query expression, and the records that hit the range will be updated or deleted at one time. This method does not require querying __rowid__ first. As long as you can clearly describe "which records need to be modified" using a query statement, you can directly initiate the operation.

Note that query-based operations only work if the conditions match data that's already been indexed. Because indexing happens asynchronously after ingestion, you'll usually need to wait until the data is queryable before running updates or deletes—so this approach doesn't work well for "modify right after write" scenarios.

Also, each query-based update or delete can affect up to 10,000 rows at most. We recommend breaking large jobs into smaller batches based on business dimensions—like billing cycle, batch, user, instance, or time window—to keep the scope of each operation manageable.

IV. Typical Scenarios

Overall, business scenarios suitable for using update/delete on a Logstore usually have the following features:

Data has typical log features. Data is continuously generated, large in size, and mainly written by append. It requires retrieval and analysis. At the same time, there are relatively low-frequency requirements for modification, backfill, or cleanup.
There is a natural time recency between modification and writing. For example, user feedback arrives with a delay, models are recomputed by version, and billing cycles are adjusted at the end of the month.
The range of each modification is relatively controllable. The hit range is naturally limited by request_id, batch, or billing cycle.

You can look at several typical scenarios below.

Risk Control and Marketing: Purge of History Labels After Rule Upgrades

Consider a risk control scenario as an example: each transaction written to a Logstore may carry a threat level computed by the model at that time. When the risk control team releases a new version of the scoring rules, they may find that the old rule underestimated the risk for certain transactions. In that case, they need to apply the new rule's logic to correct a batch of historical labels.

For instance, you could update a merchant's risk_level from medium to high for the past 7 days, ensuring that downstream review queues, threat dashboards, and reports immediately reflect the updated standard.

client.update_logs(
    project="risk",
    logstore="transactions",
    from_time=seven_days_ago,
    to_time=now,
    query='merchant_id: "M-2049" and risk_level: "medium"',
    log_item={
        "risk_level": "high",
        "risk_model_version": "v3.2",
        "re_evaluated_at": now,
    },
)

Similar scenarios also include marketing channel attribution adjustments, experiment grouping strategy fixes, report standard field purges, and content tags or user persona recomputations. Their similarity is: the rule is upgraded, and the history results also need to be purged accordingly.

Metering and Billing: Detail Correction After Billing Cycle Settlement

Metering and billing details are continuously generated by billing cycle and written to a Logatore for aggregation and reconciliation. During settlement at the end of the month, some records may need to be corrected because of discount write-offs, refunds, customer negotiations, or manual adjustments.

These types of scenarios are naturally suitable for update/delete: adjustments usually occur after the billing cycle ends, and there is a clear time recency between writing and modification. The correction range can also be delineated by fields such as billing cycle, user identifier (ID), and instance ID, with clear borders.

For example, you can delineate a batch of records by billing_period="2026-05" and instance_id="inst_8821", and only update fields such as amount, adjust_reason, and adjusted_at.

The adjustment result is directly reflected on the original details, and downstream reports, aggregated bills, and audit queries automatically see the corrected data.

LLM/Agent: Field Backfill for Delayed Feedback

A single request of an LLM/Agent application generates a large amount of procedure data. Among them, execution details, such as each step and each tool call, are suitable to remain append-only. However, information that arrives with a delay after the request ends is very suitable to be backfilled to the original request record.

This type of information includes user upvotes and downvotes, operational badcase annotations, offline quality scores, and manual review conclusions.

The invocation method is similar to the previous one. You can locate the original request record by request_id + time range, and backfill the feedback field. In this way, trace details still aggregate execution details by trace_id, and feedback analysis and quality assessment can directly query the fields on the request record.

O&M/Management Interface: Precise Modification of Records of Specified Rows

In some scenarios, modifications are triggered by humans on the interface—customer service changes the order status from "Pending" to "Contacted" in the ticketing system, the data administrator views the collected annotation data and corrects a fault label, or operations add review comments to an LLM/Agent request in the badcase management backend.

The typical flow of this type of operation is "query first, modify later": the interface queries a batch of records through a query, and these records will all carry a unique __rowid__. After the user selects one of them to modify, it is precisely located and written back by __rowid__ when it is saved.

# After the user selects a record to modify, it is precisely written back by __rowid__
client.update_logs(
    project="ops",
    logstore="work-orders",
    rowid="1|1048576|63",   # The __rowid__ retrieved from the query result
    log_item={
        "status": "contacted",
        "handler": "alice",
        "handled_at": now,
    },
)

This method is suitable for scenarios such as single manual correction, review write-back, and backend management operations. Compared with redefining conditions based on business fields, __rowid__ can precisely locate the record that the user sees on the interface.

V. Final Thoughts

Let's return to the original question: can logs be modified?

If we're talking about traditional O&M logs—records of what the system did and when errors occurred—the answer remains no. Immutability is precisely what gives them their value.

However, a Logstore today handles far more than that. Increasingly, business data is generated continuously in log form. This data needs the elasticity, retrieval, analysis, and consumption/delivery capabilities that a Logstore provides—while also giving rise to backfill, correction, and cleanup needs within business workflows.

For this type of data, native update/delete support offers a direct solution: no need to work around it through external systems, and no need to sacrifice the Logstore's inherent strengths.

If your requirements match the features described earlier, you can set enableModify=true for your Logstore to unlock more scenario possibilities.

For specific activation methods, API parameters, restriction checklists, and billing instructions, you can refer to the official document Modify and delete log data.

Enabling Boundless Intelligent Collaboration: UModel Now Open Source, Launching the Universal Semantics Standard Initiative

ObservabilityGuy — Fri, 07 Aug 2026 05:34:38 +0000

Let data speak the same language and enable boundless intelligent collaboration.

Hangzhou, May 20, 2026 — At the 2026 Alibaba Cloud Summit, Alibaba Cloud officially open sourced Unified Model (UModel) and launched the Universal Semantic Standard (USS) industry initiative, aiming to break down the fundamental barriers of semantic fragmentation in enterprise data and build a semantic foundation for large-scale AI implementation.

Every system in an enterprise faithfully records facts. The alerting system records anomalies, the log platform saves fault samples, the Trace system records invocation chains, Kubernetes manages running status, the release systemrecords changes, the Configuration Management Database maintains configurationrelationships, CRM tracks customers, and ERP manages orders and inventory.However, enterprise digitalization and intelligent transformation faces a long-underestimated fundamental barrier — semantic fragmentation.

Pervasive data silos: According to industry research, enterprises use more than 30 Software as a Service tools and internal systems on average.Data is scattered across hundreds of "data chimneys" such as O&M monitoring, business systems, financial platforms, and customer management. A "checkout-service error rate spike" may simultaneously be associated with a recent release, a downstream dependency timeout, abnormal restarts of three pods, and an SLO violation — these clues are scattered across five or six systems with no unified object border, relationship structure, or evidence chain between them. Engineers rely on experience to jump between systems and piece together context, spending over 80% of their time "finding info" rather than "making judgments."
Semantic inconsistency as a pervasive problem: For the same "sales" indicator, finance, operations, and e-commerce each have their own set — three numbers, three meanings, and three definitions sharing the same name. The same problem is equally common in the O&M realm: different monitoring platforms use different collection methods, formulas, and time windows for "error rate." Cross-platform troubleshooting often becomes a semantic translation marathon. To snap the different expressions of the concept "customer" across three systems (account, buyer, and customer_id), data analysts may need to spend two days on caliber mapping before they can begin actual analysis work.
Large-scale AI implementation blocked: Without unified semantic context, AI agents cannot reliably understand the meaning of data across platforms and frequently hallucinate, significantly undermining intelligent decision-making quality. In the past, human experience could bridge this semantic gap at the cost of low efficiency but controllable results. However, when enterprises begin involving AI agents in O&M, customer service, analysis, and automated decision-making, the gap shifts from an "efficiency problem" to a "capability problem" — an agent can invoke ten tools and retrieve ten sets of data, but cannot determine whether they belong to the same service, the same change, or the same causal chain. Without semantics, AI cannot establish cross-system data associations and causality. It can only summarize single-source data and cannot support end-to-end intelligent decision-making.
Collaboration costs increase exponentially: Data semantics across departments, systems, and tools are disconnected, resulting in persistently high communication friction. A simple data analytics task can take 3–5 days to snap definitions, translate fields, and confirm meaning — less than 20% of the time is actually spent on analysis and decision-making.

Businesses don't lack data, and they don't lack tools. What's missing is a unified semantic runtime that makes the enterprise world understandable to people, systems, and AI. To solve the above problems, Alibaba Cloud officially open sourced Unified Model (UModel) at the Cloud Summit on May 20.

UModel: Object Graph Semantic Runtime for Enterprise AI

As the core technical solution built by Alibaba Cloud to solve the above problems,UModel is an object graph semantic runtime for enterprise AI. It uses objects and relationships to describe the enterprise world, making these descriptions queryable, authenticatable, and programmable for Agent invocation. It is not another observability tool, Configuration Management Database, or knowledge graph. It sits on top of these systems and organizes the existing facts within them into a unified object graph. By combining the corresponding capabilities, it transforms enterprise data from "being recorded separately by each system" to "being uniformly organized, queried, authenticated, and invoked around objects." For the core problems of enterprises, UModel provides clear solutions:

(I) Design Choices of UModel
The UModel approach differs from most data integration solutions. The following three design choices determine its engineering form — first define the object, make the specification verifiable, and then connect existing systems based on a stable specification.

Object First: Define borders before mapping data A common override path is "data first" — connecting data from multiple systems to the same platform and then trying to aggregate objects from it. The problem with this path is that the same business entity has different expressions in different data sources, each aggregation requires remapping, and the mapping result is unstable — the "service A" aggregated today and the one aggregated tomorrow may not be the same thing.

UModel chooses to bear the upfront modeling cost: an object is defined once, and its identity and relationship structure remain stable regardless of pod reconstruction, data source switching, or new system integration. This cost is naturally amortized across multiple scenarios—the same EntitySet can be shared by O&M agents, analysis agents, and Customer Service agents.

Specification as Code: Turning Semantic Specifications into Verifiable Engineering Assets

Most enterprises also have data dictionaries or semantic specifications—usually wiki pages or Excel sheets. The problem is: outdated documents don't throw errors, renamed fields don't trigger diff tooltips, and there's no way to verify whether two teams understand the same concept consistently. Over time, specifications gradually drift from reality and become untrusted artifacts that no one relies on.

UModel manages semantic specifications as code: model changes go through PR review, imports include schema checksum validation, and whether different implementations understand the same set of semantics is automatically verified using Conformance Cases. Specifications aren't "valid just because they're written"—they "only count when tests pass," solving the disconnect between specifications and reality at the root.

Connect, Don't Replace: No Data Migration, Just Semantic Bridges The object graph stores "who the object is, what the relationship is, and where the evidence can be found."

UModel connects to existing data sources through semantic mapping — describing "where to look up an object's indicators and where to find its logs" instead of moving data over. Data stays in place, semantically organized in UModel. Compared with aggregating all data to a unified platform (high cost, long epoch, and fragile MPS queue), the cost of connecting existing systems is far lower than replacing them. Enterprises can implement incrementally without refactoring their existing IT architecture.

(II) Technical Advantages of UModel

Object graph traversability: automatically assemble complete context along relationships This is the most core technical capability of UModel. In the object graph, each entity is connected to other entities through typed relationships: a devops.service is associated with the devops.deployment that deploys it, the devops.slo that measures it, the devops.team it belongs to, and the k8s.workload that runs it. These relationships are not diagrams drawn in documents — they are topology data queryable at runtime.

Given any object, its complete relation network can be traversed through a .topo query. The agent does not need to know in advance "which systems should be checked to troubleshoot a service failure" — it starts from the failure object and traverses the graph to find associated deployments, changes, SLO violations, and downstream dependencies. The causal chain emerges along the graph structure, without relying on prompt engineering guesses.

Link system: turning "where is the data" into a property of an object UModel doesn't just define "what objects exist" — it connects objects and their evidence completely through the Link system. DataLink connects entities with observed data: a devops.service is associated with its indicator set, log set, and trace set through DataLink, and describes how to locate specific monitoring data from the entity through field mapping (for example, entity field service_id corresponds to indicator label monitored_service_id). StorageLink connects datasets with physical storage: which MetricStore the indicators are stored in and which LogStore the logs are stored in are described by StorageLink — this makes "where the data is" part of the object graph instead of being scattered in configuration files or in the memory of O&M engineers. EntitySetLink defines the topology relationship semantics between objects.

The three types of Links combine to make the object graph a complete semantic description of "who the object is, where the evidence is, and how to query the data." After obtaining an object, an agent can follow Links to find the query paths and storage positions of its indicators, logs, and traces — even if the agent currently needs to execute queries against the corresponding data sources itself, it at least knows where to query and what conditions to use.

Vendor-neutral, specification-first: the semantic layer is not bound to any platform
Neither the semantic definition nor the runtime services of UModel are tied to a specific vendor or platform:
GraphStore Provider abstracts the storage backend — currently providing three implementations: memory, file.memory, and local.ladybug. Enterprises can develop their own providers to connect to existing graph databases or storage systems.
Multi-domain coexistence — Models for DevOps, Kubernetes, and business systems can be defined in different domains of the same workspace and associated through cross-domain EntitySetLink. In the previous traversable object graph example, devops.service → k8s.workload is a cross-domain relationship.
Model definitions and public contracts (OpenAPI, MCP schema, and software development kit types) are standard format files that do not depend on specific toolchains.

This design allows enterprises to adopt incrementally: first define the model for one realm, authenticate the value, and then extend to other realms — the evolution of the semantic layer is not affected by underlying storage changes or platform switching.

(III) Umodel Evolution Route Planning
This open source release and initiative is just the first step for UModel. The following capabilities have been internally authenticated and will be gradually available in subsequent versions:

USearch:unified semantic search. The current Query Service provides structured object graph queries (.umodel, .entity, .topo), suitable for exact access to known object types and IDs. USearch complements the ability to quickly locate entities based on entity fields — supporting inverted index exact match, vector semantic approximate search, and hybrid retrieval. When an Agent faces blur clues (a service name fragment, a fault description), USearch allows it to quickly converge to a specific entity, then expand the context through the object graph.
Platform as a Service API: object-level semantic query. The open source version of Query Service queries the object graph itself (model, entity, topology). The Platform as a Service query proxy solves the next problem: given an entity, how to obtain its associated data? Based on the mapping between DataLink and StorageLink, it translates object-level intents such as "query checkout-service fault rate indicators" into executable search statements — but is not responsible for execution. The returned search statements are executed by the AI Agent or upper-layer system in the corresponding data source.
More realm models and providers: Continuously expand model packages across realms such as databases, MSMQ, Cloud Products, network devices, and business systems. At the same time, more GraphStore Provider implementations are available to support enterprises in persisting object graphs to the Graph Database or storage backend of their choice.

(IV) Administration Models and Participation Paths
UModel is now officially open source: the core closed loop can run locally, semantic specifications and Query Service are stable, and AgentGateway / MCP is active. Production-level graph storage providers, broader realm model coverage, and large-scale performance authentication are the directions that need to be advanced by the community together — which also means now is the most effective participation window to influence the project direction.

1 minute:Star the project and follow the latest developments.
5 minutes:Try the demo. Pull the code and execute make quickstart. Open the Web UI to browse example object graphs and get an intuitive understanding of the solution capabilities.
Half a day: Make your first commit. Read the model definitions in examples/quickstart multidomain/→ Define 3 EntitySets for a realm you are familiar with → Add relationships and sample data → Submit a PR.

Ongoing contribution directions (see project documents for detailed contribution guidelines):

Realm model: Contribute EntitySet and relationship definitions for realms such as databases, MSMQ, Cloud Products, and business systems.
GraphStore Provider: Connect to a new graph storage backend to drive production-grade storage capabilities
Agent tools: Contribute MCP tools, query templates, and prompts to reduce agent exploration cost
Conformance Case:Write consistency authentication use cases to ensure different implementations understand the same set of semantics

(V) Quick Start
Quick Start

This open source release provides a complete closed loop that can run locally: define the model → import workspace → write entities and relationships → unified query → agent exploration → consistency authentication. The repository contains the full test suite (contract / integration / e2e / golden tests) constraining the public contract.

Startup

make quickstart

Three portals are active after startup:

API: localhost:8080 (REST + MCP)
Web UI: localhost:5173 (browse object graph, execute queries, view agent tool list)
Preloaded data: demo workspace with multi-domain model examples including DevOps, Kubernetes, and supply chain

Developer Perspective: Command-Line Interface (CLI) and Web UI

Query the object graph with the command-line interface (CLI):

# What does the checkout-service object look like?
umctl query run demo \
  ".entity with(domain='devops', name='devops.service', query='checkout') | limit 10"
# What objects is it directly related to?
umctl query run demo \
  ".topo | graph-call getDirectRelations([(:\"devops@devops.service\" {__entity_id__: '10000000000000000000000000000101'})]) | limit 10"

You can also open the Web UI directly — browse object types and instances in the Explorer view, interactively execute the above queries in the Query view, and visualize the object relationship topology in the Graph view. For complete query syntax, see the document. Agents do not need to handwrite these statements when invoking through MCP.

AI Deep Scenario: Agent Systematic Troubleshooting Through MCP + Runbook

The checkout-service example above is single-cause, interference-free — the agent can find the answer by walking a few steps along the graph. However, failures in real production environments are often more complex: the cause is not a single change but a superposition of multiple changes, misleading clues interfere with judgment, and key evidence is scattered across different systems managed by different teams. When there are more than 5 clues involving more than 3 management domains, pure graph traversal is not enough — you need a structured diagnostic protocol to ensure the completeness and correctness of the troubleshooting.

Scenario: 02:17 AM, payment-gateway P99 latency breaches SLO

The on-duty SRE is woken up by an alert. payment-gateway is a platinum-level service — every minute of abnormal behavior means direct business loss. A quick determination is needed: roll back the most recent deployment? Scale out? Or is there another cause?

Timeline:

T-24h: The retry configuration of checkout-service was changed from 2 to 5 (a routine Configuration Change; traffic was normal at the time and no issues occurred)
T-4h: The 618 promotion was activated, and traffic began climbing to 3.5x
T-0: Alerting triggered, P99 latency > 2000ms

What makes this difficult? The root cause is not a single management event, but the superposition of two independent changes within a specific time window: the Configuration Change alone would not cause issues (4000 × 2.5 = 10,000 QPS, within capacity); the promotion traffic alone would not cause issues either (4000 × 3.5 = 14,000 QPS, acceptable). But when both take effect simultaneously: 4000 × 3.5 × 2.5 = 35,000 QPS, 8.75 times the normal capacity — payment-gateway is overwhelmed.

Trickier still: there was a deployment 12 hours ago. An SRE's first instinct is to suspect it—"what changed recently" is a cognitive shortcut for troubleshooting. But this time it's a red herring: the change was a log format adjustment, unrelated to latency. If you intuitively rollback this deployment first, the problem won't be resolved, and you'll have wasted valuable time.

Moreover, key clues belong to different management domains: alerting and deployment records are in the Platform domain (managed by the O&M team), while promotions are in the Business domain (managed by the business team). In traditional troubleshooting, SREs may not even know a promotion is underway—until they call the business team.

payment-gateway (degraded, platinum SLO)
  ← calls ← checkout-service
               ← affects ← cfg-checkout-retry (max_retries 2→5, 24h ago)
                              ← triggers ← 618 Flash Sale (3.5x traffic)
Exclusion: payment-gw v3.2.1 (12h ago, trivial logging change)
Root cause: 4000 × 3.5 × 2.5 = 35,000 QPS → 8.75x overload

UModel's answer: Runbook—a diagnostic protocol executable by agents

UModel solves such complex scenarios through Runbook. A runbook is not a document or a wiki—it is a structured protocol that agents can execute programmatically, comprising three layers:

Observation (what to check): Defines the specific steps for each check item—which entities to query, which relationships to traverse, and which fields to compare.
Conclusion (how to judge): Define matching conditions for each observation—what conditions are met to draw what conclusions, and what the severity level is.
Knowledge (why it is a problem): Provides interpretations and compute formulas for failure patterns to help the Agent understand rather than just match. Unlike retrieval-augmented generation, which allows the Agent to infer on its own after retrieving documents (prone to omissions and non-reproducible), Runbook ensures two things: completeness—it will not forget to check the key factor of service traffic; determinism—the same data, executed at different times by different models, yields consistent conclusions.

In this scenario, the platform.service.ops Runbook defines three observations for the Agent: upstream_retry_amplification (check upstream retry configuration), recent_deployment_correlation (exclude or confirm recent deployment), and business_traffic_pressure (detect service traffic pressure). The Agent executes them one by one according to the protocol and does not skip subsequent checks just because there was a "recent deployment."

From UModel to Universal Semantic Standards: Something No Single Enterprise Can Accomplish Alone

Alibaba Cloud serves more than 4 million enterprise customers worldwide and has deep product expertise in observability, data governance, and enterprise intelligence. Through years of serving enterprise customers, we have come to recognize the systematic barriers that semantic fragmentation poses to unlocking data value and implementing AI. Enterprise semantics spans too many realms, too many scenarios, and too many existing systems — O&M has its own object models, business has its own entity definitions, and each industry has its own domain knowledge. This is by no means something a single company can accomplish alone. We firmly believe that unification and openness are the only keys to unlocking the full value of enterprise data and the full potential of AI.

Therefore, alongside the open source release of UModel, Alibaba Cloud officially launches the Universal Semantic Standard (USS) industry initiative — an open source collaboration schedule for the entire industry that aims to fundamentally solve the "semantic dialect" problem of enterprise data and reshape the way humans interact with data and data interacts with applications.

We have joined hands with the first batch of founding members, including industry benchmark enterprises such as Chanjet, Shenzhou Shanglong, XPeng, Zhuoyu Technology, and JLCPCB, as well as industry institutions such as CAICT and the Chinese Academy of Sciences, covering ecosystem partners in cloud computing, observability, BI analysis, enterprise software, and other fields, to jointly lay the foundation for building a more open, interconnected, and intelligent enterprise data ecosystem. Through the USS initiative, we aim to work with partners to confront the core obstacle to artificial intelligence in the enterprise domain — the lack of unified semantic standards. This is an industry collaboration that transcends competition, with the goal of jointly solving the fundamental problem of enterprise data intercommunication and mutual recognition.

(I) Core goals and principles of the Universal Semantic Standard initiative
The core objectives of the USS initiative are: to establish a set of common, vendor-neutral enterprise-level semantic model specifications that provide unified semantic representation for all types of enterprise data, including indicators, logs, traces, management events, and business entities; to enable seamless semantic interoperability among enterprise business software, O&M tools, and AI agents; and to accelerate digital transformation and large-scale AI implementation across industries. By providing unified, standard semantic specifications, the initiative ensures that all enterprise data is defined consistently and value-aligned across platforms, departments, and systems, completely eliminating semantic ambiguity across tools, departments, and systems. The initiative follows five core principles:

Standardization: Establishes a unified enterprise semantic model definition language and data structure, covering core elements such as business entities, indicator definitions, log fields, trace labels, and alerting rules, ensuring consistency and interpretability across tools and systems.
Interoperability: Facilitates seamless data exchange and cross-platform application of semantic models among diverse enterprise systems such as application performance management, log platforms, artificial intelligence for IT operations, BI tools, CRM, and ERP.
Scalability: The semantic model supports flexible extension and customization, meeting evolving requirements from monolithic applications to cloud-native microservices, from traditional IT to large-scale distributed systems, and from general business to industry-specific scenarios.
Open source collaboration: Driven by the open source community, encouraging global developers, vendors, and enterprises to participate and contribute, ensuring the frame maintains sustained vitality and relevance amid rapid technological iteration.
Domain-specific modeling: Provides standardized semantic representations for typical enterprise scenarios—such as O&M golden signals, SLO/Service-Level Agreement definitions, business indicator systems, customer entity models, and supply chain relationships—simplifying the integration and reuse of multi-source data.

USS fully respects and is compatible with existing excellent industry standards. It is not intended to override them, but to provide a unified semantic framework at a higher abstraction layer. OpenTelemetry defines the naming conventions for traces/metrics/logs in the observability realm but does not cover business data semantics—USS extends upon this foundation to cover all enterprise scenarios. Open Semantic Interchange (OSI) addresses the problem of semantic model exchange between BI/analysis platforms—USS collaborates and complements it, covering a wider range of enterprise application scenarios. The Schema.org/W3C Semantic Web standards provide common semantic tagging and ontology modeling capabilities—USS draws on their design philosophy but focuses on the practical requirements of an enterprise's internal data ecosystem. USS serves as a semantic unified layer for all enterprise scenarios, bridging downward to domain-specific proprietary standards while providing a unified semantic context across systems for AI agents.

(II) Expected Results
Through the USS initiative, we will jointly achieve:

Accelerate enterprise AI scaling: A unified semantics standard enables AI models to truly "understand" the meaning of data from different platforms and departments, significantly improving the accuracy and deployment speed of AI applications across all scenarios, including artificial intelligence for IT operations, intelligent customer service, intelligent analysis, and intelligent prediction.
Reduce data governance costs by an order of magnitude: A universal semantics standard provides a common language for enterprise data architectures spanning multiple data sources, tools, and systems, freeing data teams from tedious metric snap and field translation so they can focus more on data value mining.
Ensure vendor neutrality and freedom of choice: The frame is independent of any specific platform or AI tool vendor, giving enterprises true freedom of choice when building digital infrastructure and avoiding vendor lock-in.
Build an enterprise-grade "semantics operating system": Upgrade from a passively consulted "data dictionary" to a live, proactive semantics runtime that can be programmatically invoked by AI agents, laying the foundation for future large-scale multi-agent collaboration across enterprises.

Conclusion

At this pivotal moment when AI is penetrating enterprise production systems at scale, semantic unification of data has become the core foundation for industrial upgrade. The UModel and USS initiatives are our answer for this era — ensuring data is no longer scattered fragments and enabling AI to truly understand how enterprises run.

The core philosophy of UModel is clear and resolute: reconstruct an enterprise's business and data worlds into a unified object graph, enabling AI to understand the complete context along the relationships between entities and make reliable judgments based on traceable evidence. As the capability boundaries of agents continue to expand, what truly constrains the release of their value will no longer be the upper limit of the model itself, but whether the enterprise's business world can be truly understood and perceived by AI. The mission of UModel is to bridge this critical semantic gap — so far, we have taken the lead in completing the implementation and validation of this path in the observability realm, proving its feasibility.

Achieving this goal is by no means something a single team can accomplish alone. That is why we chose to open source the project and launch the Universal Enterprise Semantic Standards initiative — we firmly believe that the right path for this undertaking is rooted in the spirit of open collaboration, and we uphold three open principles: open standards and specifications, open technology implementation, and open effect verification. Starting from observability scenarios, we have completed the initial implementation of our methodology. Now, we look forward to working hand in hand with community partners to gradually extend the capabilities of the enterprise object graph from this realm to the broader landscape of data governance, business systems, and cross-domain collaboration.

We sincerely invite every industry peer — whether you are an enterprise software vendor, an AI solutions provider, a developer in the open source community, or an enterprise user across industries — to join us on this pivotal journey to build a more open, more connected, and more intelligent enterprise data ecosystem. We believe that through open collaboration, we will jointly drive the transformation of enterprise AI from "fragment summarization" to "deep understanding," and together create a truly AI-perceptible intelligent enterprise future.

Let data speak the same language. Let intelligence collaborate without boundaries.

Alibaba Cloud Launched a Global Intelligent O&M Platform from "Passive Fire Fighting" to "Active Autonomy"

ObservabilityGuy — Mon, 06 Jul 2026 05:23:52 +0000

This article introduces Alibaba Cloud's STAROps, an AI-native intelligent operations platform that leverages autonomous agents to transition IT manage...

On May 20, Alibaba Cloud officially released the AI-native global intelligent O&M platform STAROps.

The platform uses large model and agent technology as the core engine and Alibaba Cloud observable product system as the data base to deeply integrate cross-domain observable data with large language model reasoning capability. Users only need to define operation and maintenance objectives in natural language, and operation and maintenance agents can independently complete the full closed loop of dynamic planning, safe execution and result verification.

STAROps is designed around four capability dimensions: Sense global perception, Target goal orientation, Autonomy autonomous operationand maintenance, and Resilience business continuity. It provides three core functions:

Intelligent Assistant directly converts natural language into unified query and diagnosis results of cross-domain observation data. Alert analysis, metric interpretation, and log diagnosis are completed in one dialogue window without multi-platform switching.
The long-term task mechanism allows Agent to take over high-frequency repetitive work such as inspection, alarm analysis, periodic reporting, etc., and can independently execute cross-day or even cross-month asynchronous operation and maintenance plans once aligned with predefined objectives.
Digital employees enable enterprises to build dedicated SRE intelligence for each team, customize responsibilities, permissions, and tool sets, and solidify expert experience into role rules at one time.

At the technical architecture level, the competitiveness of STAROps is reflected in four dimensions.

Unified Observability Data

Unifies logs, metrics, traces, events, topology, and changes with PB-scale daily ingestion, EB-scale storage, low-latency analysis, multi-AZ deployment, and 99.95% reliability.

Operational Digital Twin

Builds a unified graph model (UModel) from entities, relationships, observability data, and operational knowledge, helping agents understand systems, trace blast radiuses, and reason about root causes in a shared context.

AI Analytics Operators

Supports anomaly detection, log clustering, trace analysis, performance profiling, and change analysis, reducing the cost of processing massive raw data while improving diagnostic efficiency and result stability.

Continuous Improvement Flywheel

Builds a realistic evaluation loop with simulation, fault injection, diagnostic assessment, and feedback, creating a measurable, roll-back-ready system for continuous agent improvement.

The essence of cloud computing lies in orchestrating computing resources as a service in an efficient way, and what STAROps is doing is extending this principle to operations and maintenance. Manpower-intensive O&M tasks are intelligently performed by using agents to schedule large-scale O&M operations. The digital employee mechanism of STAROps provides enterprises with this progressive path: it not only supports embedding AI in existing processes to improve efficiency, but also supports building a new agent native O&M mode.
**
In terms of access form, STAROps provides a variety of access solutions such as **OpenAPI and MCP integration, page embedding, and mainstream IM access. Enterprises can release value in existing workflows at the lowest migration cost. The built-in manual approval mechanism of the platform ensures that key decision nodes are still under manual control, striking a balance between the efficiency of agent independent execution and security compliance.

Along with the product release, Alibaba Cloud synchronizes the open source UModel unified data model project with the RCA-100 evaluation benchmark set, and jointly launched the "Enterprise Common Semantic Standard Industry Initiative" with more than 10 industry partners and academic institutions such as the Institute of Information and Communications Technology, Xiaopeng Automobile, and the Software Institute of the Chinese Academy of Sciences.

Currently, STAROps has been officially launched on the Alibaba Cloud official website. As AI reshapes every aspect of software development, O&M, as the last line of defense to ensure business resilience, is ushering in a paradigm transition from tool assistance to agent autonomy. Alibaba Cloud uses STAROps as a starting point to push Agentic Ops from concept to production-level implementation.

From API to AI Agent: Alibaba Cloud CloudMonitor Command-line Interface (CLI) + Agent Skill in Practice

ObservabilityGuy — Thu, 02 Jul 2026 10:14:34 +0000

This article introduces the Alibaba Cloud CloudMonitor CLI and Agent Skill to enable AI agents to automate observability O&M workflows.

You can use the official CloudMonitor CLI + Agent Skill to allow an AI Agent to securely execute observability O&M jobs.

30 Second Overview

Alibaba Cloud CloudMonitor CLI (aliyun cms2) consolidates capabilities such as integration, configuration, queries, alerting, and management events in the Cloud Monitor Service (CMS) 2.0 console into a unified command line entry. CMS Agent Skill organizes these commands into business workflows for AI Agents.

In the past, O&M automation often started from APIs: querying documents, assembling parameters, writing scripts, and invoking APIs. Now, by using the CloudMonitor CLI + Agent Skill, these capabilities can be organized into standardized workflows that AI Agents can understand, execute, and validate.

For O&M engineers, its value is not "one more tool", but allowing you to describe O&M targets in natural language, and having the AI Agent complete scenario understanding, CLI invocations, API executions, and result validations, transforming repetitive, multi-step, and error-prone observability O&M operations into confirmable, auditable, and reusable automated flows.

Why You Need CLI + Agent Skill

With the continuous growth of business size and infrastructure on the cloud, observability O&M runs through the end-to-end flows of resource integration, metric/log collection, alerting administration, link troubleshooting, Root Cause Analysis, and stability operations, and the O&M workload and operation complexity rise accordingly. At the same time, relying on powerful language understanding and job orchestration capabilities, AI Agents are becoming a new O&M collaboration entry. More and more teams are starting to try handing over repetitive, standardized, and multi-step jobs to Agents for assisted execution, and handing over complex problem troubleshooting to AI for assisted analysis.

However, to allow AI Agents to truly enter the production O&M closed loop, they cannot just stay at the stage of "understanding problems, generating suggestions, or scripts". A stable execution entry for CloudMonitor capabilities, standardized realm flows, necessary manual confirmations, and verifiable execution results are also required. The CloudMonitor CLI + Agent Skill is exactly the capability suite built for this requirement.

CLI + Skill Solutions

Alibaba Cloud CloudMonitor CLI (aliyun cms2) provides a unified, stable, and auditable capability entry. CMS Agent Skill consolidates the business semantics and operation flows in the CloudMonitor realm into workflows that AI Agents can understand and execute. By coordinating the two, AI Agents can start from natural language instructions such as "help me integrate this Container Service for Kubernetes (ACK) cluster into CloudMonitor", and automatically complete scenario detection, parameter generation, CLI invocations, API executions, and result validations.

Unified command tree:
The CLI has covered capabilities such as Integration Center, Prometheus service, application monitoring, Real User Monitoring, alerting center, and Event Center in the CMS 2.0 console. In the future, it will continue to cover capabilities such as Synthetic Monitoring and Grafana dashboards to achieve complete coverage of the CMS 2.0 console.
Native adaptation for AI Agent:
- It provides standardized, clear, and detailed --help information, and supports auxiliary capabilities such as --show-schema and --show-example-body to help AI accurately process various business scenarios.
- By default, it uses -o text to output compact Comma-Separated Values (CSV), significantly reducing AI token consumption.
- Through structured JavaScript Object Notation (JSON) error codes, it supports Agents in automatically making decisions and repairing based on fault reasons.
Skill driver:
The supporting Skill documents consolidate complete business workflows, allowing Agents to complete complex multi-step operations without hard coding.

CLI + Skill Workflow

For O&M engineers, the most intuitive change is: instead of starting operations from console entries or API parameters, they start from a clear O&M target, and the Agent completes subsequent executions and validations according to standard flows. The core of this link is "controllable automation": the Agent will not bypass the O&M system, but executes operations through the unified CLI entry and the business rules consolidated in the Skill. This can not only reduce repetitive labor, but also retain necessary permissions, confirmations, and audit borders.

Installation and Configuration
Install Skill/CLI

You can open the alibabacloud-cms-manage Skill on the Alibaba Cloud Agent Skills portal and follow the instructions on the interface to install the Skill.

After the installation is complete, when the AI Agent uses the Skill, the AI Agent automatically detects and guides you to install or update the Alibaba Cloud command-line interface (CLI) and the cms2 plugin to the required version. You do not need to manually handle environment dependencies.

# Verify that the CLI installation succeeded
aliyun version
# Verify that the cms2 plugin is active
aliyun cms2 --help

Configure Credentials
Multiple credential types such as AccessKey and Security Token Service token are supported. For more information, see Configure identity credentials for Alibaba Cloud CLI.

# Interactive configuration (recommended for first-time use)
aliyun configure

# Non-interactive configuration
aliyun configure set \
--access-key-id YOUR_AK \
--access-key-secret YOUR_SK \
--region cn-hangzhou

Practical Scenario 1 (Integration Center): Integrate Container Service for Kubernetes (ACK) Clusters into CloudMonitor

Business Scenario
The Site Reliability Engineering (SRE) team created an ACK cluster to deploy microservices. The SRE team needs to integrate the metrics of the cluster, such as nodes, pods, and containers, into CloudMonitor.

Usage
You only need to enter the following text in the AI Agent conversation:

Help me check which container clusters in Hangzhou do not have observability capabilities, and help me integrate them.

The Agent automatically completes the entire integration flow. Users only need to confirm at key steps.

Core Capabilities Supported by the AI Agent

Common Scenarios and Prompt Samples for Integration Center
Integrate by resource group: You can integrate all Relational Database Service (RDS) instances in the Beijing area under the default resource group into {workspace} of CloudMonitor.

Integrate by label: You can integrate all Elastic Compute Service (ECS) instances that match the label key={tagKey} and value={tagValue} into {workspace} of CloudMonitor.

Integrate across accounts: You can integrate all AI gateways in the Shanghai area of {resource directory member accounts uid} into CloudMonitor.

Monitoring widget deployment: You can add the integration of the ACK cost Insight widget in the integration policy {policy id/name}.

Metric collection target check: You can check whether the apiserver-related scrape targets of the ACK cluster {cluster Id/name} are normal.

Custom collection rule query: You can query the serviceMonitor/podMonitor/customJob list of the integration policy {policy id/name}.

Practical Scenario 2 (Alerting Center): Intelligent Alert Rule Management

Business scenario
The SRE needs to establish a comprehensive alerting system for the production environment. For example, the SRE configures professional node alert rules for container service cluster nodes.

Usage
The following is a typical conversation sample:

What recommendations do you have for container alerting? Then help me apply them.

Core Capabilities Supported by the AI Agent

Common Scenarios and Prompt Samples for Alerting Center
Intelligently analyze alert rules: You can analyze whether the existing alerts are configured reasonably and whether alert noise exists. If the configuration is unreasonable, you can modify the configuration with one-click.

Query alert rules: You can query all running alert rules of cloud service monitoring in the workspace {workspace}.

Modify Alert Rule contacts: You can change the Notification Recipient of the Alert Rule {rule ID/Name} to {contact}.

Delete an Alert Rule: You can delete the {rule Name} Alert Rule of the Prometheus instance {instance ID/Name}.

Query alerting history: You can query the alerting history of the Alert Rule {rule ID/Name} within 1 week.

Practical Scenario 3 (Prometheus service): Prometheus Instance Management and Data queries

Business Scenario
The O&M team needs to manage multiple Prometheus instances, analyze metrics and business health status, and configure Recording Rules to pre-aggregate high-frequency metrics.

Usage
The following are typical dialogue samples:

Help me check which Prometheus instances are available in Hangzhou, and group them by workspace.

The Following Is an Overview of the Core Capabilities Supported by the Agent

Common Scenarios and Prompt Samples for the Prometheus Service
Modify the storage duration of a Prometheus instance: You can modify the storage duration of the Prometheus instance {instance ID/Name} to 90 Days, and the archive duration to 180 Days.

Create a Recording Rule: You can create a Recording Rule under the Prometheus instance {instance ID/Name} to pre-aggregate the 5 minute average CPU utilization of each edge zone.

Stop a Recording Rule: You can stop the {aggregation Job Name} pre-aggregation Job under the Prometheus instance {instance ID/Name}.

Create a Prometheus aggregation view: You can create an aggregation view {aggregation view Name} that contains all Prometheus instances in the {area Name} area under the {workspace} Space.

Practical Scenario 4 (Application Performance Monitoring (APM)): Application Monitoring/AI Observability Onboarding

The onboarding flow for this scenario includes steps such as initializing the APM infrastructure, obtaining Credentials, registering the application, obtaining the configuration template, and authenticating the onboarding. The traditional onboarding procedure is relatively complex. Using the command-line interface (CLI) + Skill can greatly simplify the flow and achieve natural language interactive onboarding.

Practical scenario 5 (Data query): metadata, Prometheus Query Language (PromQL), and CloudMonitor Basic metric queries

Business Scenario
You can query metadata, Prometheus metric Data, and CloudMonitor Basic metric Data to analyze business running conditions and troubleshoot faults or problems.

Usage
The following are typical dialogue samples:

List of ECS instances with the highest CPU utilization: You can find the 10 ECS instances with the highest CPU utilization in the last half hour.

The Following Is an Overview of the Core Capabilities Supported by the Agent

Common Scenarios and Prompt Samples for Data Queries
RDS slow queries: You can query the Quantity Trend of slow queries with a running time exceeding 1 second in the past 30 minutes.

Waste of container resource Requests: You can find "zombie" resources in the container cluster that have overly large resource requests but very little actual usage in the past 7 Days.

Suspected container pod memory leak: You can find the List of pods under {ns} of the container cluster {cluster Name/ID} whose memory usage has continuously increased in the past 1 hour and whose current value exceeds 90% of the limit.

Summary

Alibaba Cloud CloudMonitor command-line interface (CLI) (aliyun cms2) and the accompanying CMS Agent Skill do not just migrate console and API capabilities to the command line, but also build a standard operation interface for AI agents for observable O&M. It unifies the capabilities scattered across scenarios such as provisioning, configuration, query, alerting, and management events. This allows O&M engineers to express targets more naturally, execute operations in a more controllable manner, and complete authentication and auditing with a clearer link.

For O&M teams, this means that observability construction is gradually moving from the manual stage of "people finding entrances, people piecing together parameters, and people performing authentication" to the collaborative stage of "people defining targets, agents orchestrating flows, CLI executing operations, and AI validating results".

AI is not intended to override O&M judgments, but to significantly reduce the costs of repetitive operations, cross-system collaboration, and complex flow execution, improve the efficiency of troubleshooting and fault localization, and allow SREs to devote more energy to higher-value work such as stability design, alerting administration, and fault review.

In the future, we will continue to enrich the capability scope of CLI and Skill, and comprehensively overwrite CloudMonitor business scenarios. In the AI era, CloudMonitor CLI and Skill hope to become a stable, trusted, and extensible observability capability base between O&M engineers and AI agents, promoting automated and Intelligent O&M from single-point attempts to large-scale implementation.

Appendix—CMS CLI Command Tree

aliyun cms2
│
│                # Integration domain
├── integration                 Integration (includes the full lifecycle of provisioning policies, add-on widgets, collection rules, etc.)
│   ├── policy                  Integration policy Management, including commands such as create, get, update, delete, and list.
│   ├── storage                 Query Prometheus storage instances attached to provisioning policies, including commands such as list.
│   ├── dashboard               Query Grafana dashboards associated with provisioning policies, including commands such as list.
│   ├── resource                Query resources of container service class provisioning policies, including commands such as list.
│   ├── job-target              Query the Status of scrape targets of Collection Tasks of provisioning policies, including commands such as list.
│   ├── service-monitor         Query Kubernetes ServiceMonitor collection rules of provisioning policies, including commands such as list.
│   ├── pod-monitor             Query Kubernetes PodMonitor collection rules of provisioning policies, including commands such as list.
│   ├── custom-job              Query Custom Prometheus Collection Jobs of provisioning policies, including commands such as list.
│   ├── addon-release           Management of deployed widget instances of provisioning policies, including commands such as create, get, update, delete, and list.
│   └── addon                   Management of active provisioning widget folders, including commands such as get and list.
├── workspace                   Workspace Management, including commands such as create, get, list, update, and delete.
│
│               # APP Application Management domain
├── prometheus                  Prometheus service Management (includes Prometheus instances, aggregation views, RecordingRules, etc.)
│   ├── instance                Prometheus instance management, including commands such as create|get|update|delete|list
│   ├── view                    Prometheus aggregation view management, including commands such as create|get|update|delete|list
│   └── recording-rule          RecordingRule pre-aggregation management, including commands such as create|get|update|start|stop|delete|list
├── apm                         application performance monitoring management
│   ├── service                 application performance management (APM) application service management, including commands such as create|get|update|delete|list
│   └── configuration           APM configuration management, including commands such as get|create
├── rum                         Real User Monitoring management
│   ├── service                 Real User Monitoring (RUM) application service management, including commands such as create|get|update|delete|list
│   └── configuration           RUM configuration management, including commands such as get|create
│
│               # alerting and management event domain
├── alert                       alerting center management (including Alert Rule, alerting template, alerting History, etc.)
│   ├── rule                    Alert Rule management, including commands such as create|get|update|patch|delete|list|enable|disable
│   ├── template                Alert Rule template management, including commands such as list|get|create|update|delete|apply
│   └── history                 alerting trigger and recovery History management, including commands such as list
├── notification-channel        notification channel management
│   ├── contact                 alert contact (Email, text message, and DingTalk) management, including commands such as list
│   ├── robot                   alerting robot (DingTalk/Lark/WeCom group robot) management, including commands such as list
│   └── webhook                 Webhook address management, including commands such as list
├── event-hub                   Event Center management, including commands such as list|get
│
│               # Data query domain
├── metric                      Metric query
│   ├── promql                  Prometheus Query Language (PromQL) instant/range query and metadata retrieval, including commands such as query|query-range|labels|label-values|series
│   └── basic                   CloudMonitor 1.0 Metric query, including commands such as points|latest|range|top|export
├── trace                       Trace data query, including commands such as search and tree
├── entity                      Cloud resource and EntityStore query, including commands such as query
└── meta                        Metadata query, including commands such as metrics, namespaces, and events

Enabling Boundless Intelligent Collaboration: UModel Now Open Source, Launching the Universal Semantics Standard Initiative

ObservabilityGuy — Thu, 02 Jul 2026 09:48:30 +0000

Let data speak the same language and enable boundless intelligent collaboration.

Pervasive data silos: According to industry research, enterprises use more than 30 Software as a Service tools and internal systems on average.Data is scattered across hundreds of "data chimneys" such as O&M monitoring, business systems, financial platforms, and customer management. A "checkout-service error rate spike" may simultaneously be associated with a recent release, a downstream dependency timeout, abnormal restarts of three pods, and an SLO violation — these clues are scattered across five or six systems with no unified object border, relationship structure, or evidence chain between them. Engineers rely on experience to jump between systems and piece together context, spending over 80% of their time "finding info" rather than "making judgments."
Semantic inconsistency as a pervasive problem: For the same "sales" indicator, finance, operations, and e-commerce each have their own set — three numbers, three meanings, and three definitions sharing the same name. The same problem is equally common in the O&M realm: different monitoring platforms use different collection methods, formulas, and time windows for "error rate." Cross-platform troubleshooting often becomes a semantic translation marathon. To snap the different expressions of the concept "customer" across three systems (account, buyer, and customer_id), data analysts may need to spend two days on caliber mapping before they can begin actual analysis work.
Large-scale AI implementation blocked: Without unified semantic context, AI agents cannot reliably understand the meaning of data across platforms and frequently hallucinate, significantly undermining intelligent decision-making quality. In the past, human experience could bridge this semantic gap at the cost of low efficiency but controllable results. However, when enterprises begin involving AI agents in O&M, customer service, analysis, and automated decision-making, the gap shifts from an "efficiency problem" to a "capability problem" — an agent can invoke ten tools and retrieve ten sets of data, but cannot determine whether they belong to the same service, the same change, or the same causal chain. Without semantics, AI cannot establish cross-system data associations and causality. It can only summarize single-source data and cannot support end-to-end intelligent decision-making.
Collaboration costs increase exponentially: Data semantics across departments, systems, and tools are disconnected, resulting in persistently high communication friction. A simple data analytics task can take 3–5 days to snap definitions, translate fields, and confirm meaning — less than 20% of the time is actually spent on analysis and decision-making. Businesses don't lack data, and they don't lack tools. What's missing is a unified semantic runtime that makes the enterprise world understandable to people, systems, and AI. To solve the above problems, Alibaba Cloud officially open sourced Unified Model (UModel) at the Cloud Summit on May 20.

1.UModel: Object Graph Semantic Runtime for Enterprise AI

(I) Design Choices of UModel
The UModel approach differs from most data integration solutions. The following three design choices determine its engineering form — first define the object, make the specification verifiable, and then connect existing systems based on a stable specification.

Object First: Define borders before mapping data A common override path is "data first" — connecting data from multiple systems to the same platform and then trying to aggregate objects from it. The problem with this path is that the same business entity has different expressions in different data sources, each aggregation requires remapping, and the mapping result is unstable — the "service A" aggregated today and the one aggregated tomorrow may not be the same thing.

Specification as Code: Turning Semantic Specifications into Verifiable Engineering Assets Most enterprises also have data dictionaries or semantic specifications—usually wiki pages or Excel sheets. The problem is: outdated documents don't throw errors, renamed fields don't trigger diff tooltips, and there's no way to verify whether two teams understand the same concept consistently. Over time, specifications gradually drift from reality and become untrusted artifacts that no one relies on.

Connect, Don't Replace: No Data Migration, Just Semantic Bridges The object graph stores "who the object is, what the relationship is, and where the evidence can be found."

2.Technical Advantages of UModel

Object graph traversability: automatically assemble complete context along relationships This is the most core technical capability of UModel. In the object graph, each entity is connected to other entities through typed relationships: a devops.service is associated with the devops.deployment that deploys it, the devops.slo that measures it, the devops.team it belongs to, and the k8s.workload that runs it. These relationships are not diagrams drawn in documents — they are topology data queryable at runtime.

Link system: turning "where is the data" into a property of an object UModel doesn't just define "what objects exist" — it connects objects and their evidence completely through the Link system. DataLink connects entities with observed data: a devops.service is associated with its indicator set, log set, and trace set through DataLink, and describes how to locate specific monitoring data from the entity through field mapping (for example, entity field service_id corresponds to indicator label monitored_service_id). StorageLink connects datasets with physical storage: which MetricStore the indicators are stored in and which LogStore the logs are stored in are described by StorageLink — this makes "where the data is" part of the object graph instead of being scattered in configuration files or in the memory of O&M engineers. EntitySetLink defines the topology relationship semantics between objects.

Vendor-neutral, specification-first: the semantic layer is not bound to any platform
Neither the semantic definition nor the runtime services of UModel are tied to a specific vendor or platform:
GraphStore Provider abstracts the storage backend — currently providing three implementations: memory, file.memory, and local.ladybug. Enterprises can develop their own providers to connect to existing graph databases or storage systems.
Multi-domain coexistence — Models for DevOps, Kubernetes, and business systems can be defined in different domains of the same workspace and associated through cross-domain EntitySetLink. In the previous traversable object graph example, devops.service → k8s.workload is a cross-domain relationship.
Model definitions and public contracts (OpenAPI, MCP schema, and software development kit types) are standard format files that do not depend on specific toolchains.
This design allows enterprises to adopt incrementally: first define the model for one realm, authenticate the value, and then extend to other realms — the evolution of the semantic layer is not affected by underlying storage changes or platform switching.

3.Umodel Evolution Route Planning

This open source release and initiative is just the first step for UModel. The following capabilities have been internally authenticated and will be gradually available in subsequent versions:

USearch:unified semantic search. The current Query Service provides structured object graph queries (.umodel, .entity, .topo), suitable for exact access to known object types and IDs. USearch complements the ability to quickly locate entities based on entity fields — supporting inverted index exact match, vector semantic approximate search, and hybrid retrieval. When an Agent faces blur clues (a service name fragment, a fault description), USearch allows it to quickly converge to a specific entity, then expand the context through the object graph.
Platform as a Service API: object-level semantic query. The open source version of Query Service queries the object graph itself (model, entity, topology). The Platform as a Service query proxy solves the next problem: given an entity, how to obtain its associated data? Based on the mapping between DataLink and StorageLink, it translates object-level intents such as "query checkout-service fault rate indicators" into executable search statements — but is not responsible for execution. The returned search statements are executed by the AI Agent or upper-layer system in the corresponding data source.
More realm models and providers: Continuously expand model packages across realms such as databases, MSMQ, Cloud Products, network devices, and business systems. At the same time, more GraphStore Provider implementations are available to support enterprises in persisting object graphs to the Graph Database or storage backend of their choice.

4.Administration Models and Participation Paths

UModel is now officially open source: the core closed loop can run locally, semantic specifications and Query Service are stable, and AgentGateway / MCP is active. Production-level graph storage providers, broader realm model coverage, and large-scale performance authentication are the directions that need to be advanced by the community together — which also means now is the most effective participation window to influence the project direction.

1 minute: Star the project and follow the latest developments.
5 minutes:Try the demo. Pull the code and execute make quickstart. Open the Web UI to browse example object graphs and get an intuitive understanding of the solution capabilities.
Half a day: Make your first commit. Read the model definitions in examples/quickstart multidomain/→ Define 3 EntitySets for a realm you are familiar with → Add relationships and sample data → Submit a PR.
Ongoing contribution directions (see project documents for detailed contribution guidelines):
Realm model: Contribute EntitySet and relationship definitions for realms such as databases, MSMQ, Cloud Products, and business systems.
GraphStore Provider: Connect to a new graph storage backend to drive production-grade storage capabilities
Agent tools: Contribute MCP tools, query templates, and prompts to reduce agent exploration cost
Conformance Case: Write consistency authentication use cases to ensure different implementations understand the same set of semantics

5.Quick Start

Quick Start
This open source release provides a complete closed loop that can run locally: define the model → import workspace → write entities and relationships → unified query → agent exploration → consistency authentication. The repository contains the full test suite (contract / integration / e2e / golden tests) constraining the public contract.

Startup

make quickstart

Three portals are active after startup:

API: localhost:8080 (REST + MCP)
Web UI: localhost:5173 (browse object graph, execute queries, view agent tool list)
Preloaded data: demo workspace with multi-domain model examples including DevOps, Kubernetes, and supply chain

Developer Perspective: Command-Line Interface (CLI) and Web UI

Query the object graph with the command-line interface (CLI):

# What does the checkout-service object look like?
umctl query run demo \
  ".entity with(domain='devops', name='devops.service', query='checkout') | limit 10"
# What objects is it directly related to?
umctl query run demo \
  ".topo | graph-call getDirectRelations([(:\"devops@devops.service\" {__entity_id__: '10000000000000000000000000000101'})]) | limit 10"

AI Deep Scenario: Agent Systematic Troubleshooting Through MCP + Runbook

Scenario: 02:17 AM, payment-gateway P99 latency breaches SLO

Timeline:

T-24h: The retry configuration of checkout-service was changed from 2 to 5 (a routine Configuration Change; traffic was normal at the time and no issues occurred)
T-4h: The 618 promotion was activated, and traffic began climbing to 3.5x
T-0: Alerting triggered, P99 latency > 2000ms What makes this difficult? The root cause is not a single management event, but the superposition of two independent changes within a specific time window: the Configuration Change alone would not cause issues (4000 × 2.5 = 10,000 QPS, within capacity); the promotion traffic alone would not cause issues either (4000 × 3.5 = 14,000 QPS, acceptable). But when both take effect simultaneously: 4000 × 3.5 × 2.5 = 35,000 QPS, 8.75 times the normal capacity — payment-gateway is overwhelmed.

payment-gateway (degraded, platinum SLO)
  ← calls ← checkout-service
               ← affects ← cfg-checkout-retry (max_retries 2→5, 24h ago)
                              ← triggers ← 618 Flash Sale (3.5x traffic)
Exclusion: payment-gw v3.2.1 (12h ago, trivial logging change)
Root cause: 4000 × 3.5 × 2.5 = 35,000 QPS → 8.75x overload

UModel's answer: Runbook—a diagnostic protocol executable by agents

UModel solves such complex scenarios through Runbook. A runbook is not a document or a wiki—it is a structured protocol that agents can execute programmatically, comprising three layers:

Observation (what to check): Defines the specific steps for each check item—which entities to query, which relationships to traverse, and which fields to compare.
Conclusion (how to judge): Define matching conditions for each observation—what conditions are met to draw what conclusions, and what the severity level is.
Knowledge (why it is a problem): Provides interpretations and compute formulas for failure patterns to help the Agent understand rather than just match.
Unlike retrieval-augmented generation, which allows the Agent to infer on its own after retrieving documents (prone to omissions and non-reproducible), Runbook ensures two things: completeness—it will not forget to check the key factor of service traffic; determinism—the same data, executed at different times by different models, yields consistent conclusions.

From UModel to Universal Semantic Standards: Something No Single Enterprise Can Accomplish Alone

Standardization: Establishes a unified enterprise semantic model definition language and data structure, covering core elements such as business entities, indicator definitions, log fields, trace labels, and alerting rules, ensuring consistency and interpretability across tools and systems.
Interoperability: Facilitates seamless data exchange and cross-platform application of semantic models among diverse enterprise systems such as application performance management, log platforms, artificial intelligence for IT operations, BI tools, CRM, and ERP.
Scalability: The semantic model supports flexible extension and customization, meeting evolving requirements from monolithic applications to cloud-native microservices, from traditional IT to large-scale distributed systems, and from general business to industry-specific scenarios.
Open source collaboration: Driven by the open source community, encouraging global developers, vendors, and enterprises to participate and contribute, ensuring the frame maintains sustained vitality and relevance amid rapid technological iteration.
Domain-specific modeling: Provides standardized semantic representations for typical enterprise scenarios—such as O&M golden signals, SLO/Service-Level Agreement definitions, business indicator systems, customer entity models, and supply chain relationships—simplifying the integration and reuse of multi-source data. USS fully respects and is compatible with existing excellent industry standards. It is not intended to override them, but to provide a unified semantic framework at a higher abstraction layer. OpenTelemetry defines the naming conventions for traces/metrics/logs in the observability realm but does not cover business data semantics—USS extends upon this foundation to cover all enterprise scenarios. Open Semantic Interchange (OSI) addresses the problem of semantic model exchange between BI/analysis platforms—USS collaborates and complements it, covering a wider range of enterprise application scenarios. The Schema.org/W3C Semantic Web standards provide common semantic tagging and ontology modeling capabilities—USS draws on their design philosophy but focuses on the practical requirements of an enterprise's internal data ecosystem. USS serves as a semantic unified layer for all enterprise scenarios, bridging downward to domain-specific proprietary standards while providing a unified semantic context across systems for AI agents.

(II) Expected Results
Through the USS initiative, we will jointly achieve:

Accelerate enterprise AI scaling: A unified semantics standard enables AI models to truly "understand" the meaning of data from different platforms and departments, significantly improving the accuracy and deployment speed of AI applications across all scenarios, including artificial intelligence for IT operations, intelligent customer service, intelligent analysis, and intelligent prediction.
Reduce data governance costs by an order of magnitude: A universal semantics standard provides a common language for enterprise data architectures spanning multiple data sources, tools, and systems, freeing data teams from tedious metric snap and field translation so they can focus more on data value mining.
Ensure vendor neutrality and freedom of choice: The frame is independent of any specific platform or AI tool vendor, giving enterprises true freedom of choice when building digital infrastructure and avoiding vendor lock-in.
Build an enterprise-grade "semantics operating system": Upgrade from a passively consulted "data dictionary" to a live, proactive semantics runtime that can be programmatically invoked by AI agents, laying the foundation for future large-scale multi-agent collaboration across enterprises.

6.Conclusion

Let data speak the same language. Let intelligence collaborate without boundaries.

Alibaba Cloud Releases RCA Benchmark, the Industry's First Open Source Root Cause Analysis Benchmark System for Agentic Ops

ObservabilityGuy — Tue, 16 Jun 2026 01:59:01 +0000

This article introduces Alibaba Cloud's open-source RCA Benchmark for evaluating AI agents in IT operations.

Alibaba Cloud has released RCA Benchmark to build a standardized root cause analysis evaluation dataset and evaluation protocol system for Agentic Ops. It is also the industry's first open source benchmark project that addresses the evaluation of AI Agent diagnostics capabilities for distributed system failures at the system level. Alibaba Cloud has partnered with institutions across the realms of observability, artificial intelligence for IT operations, and cloud-native infrastructure — including CAICT, the Institute of Software/Computer Network Information Center of the Chinese Academy of Sciences, Tsinghua University, Fudan University, and Nankai University — to jointly build an industrial ecosystem and establish a standardized, trustworthy O&M agent evaluation system, laying a solid foundation for large-scale industry adoption.

Based on long-term product implementation and service practices in the observability and artificial intelligence for IT operations realms, Alibaba Cloud has recognized that rootcause analysis is the most complex and hardest-to-standardize core process in O&M agent capability evaluation. Unlike tasks with fixed inputs and ground truth such as text Q&A and code generation, RCA Agents operate against continuously running distributed complex architectures. They must proactively filter valid info from multi-source observability data such as indicators, logs, Tracing Analysis, and system events, trace abnormal propagation paths based on service dependencies and entity topology relationships, and ultimately locate the root cause of failures. The industry has not yet established a unified, systematic evaluation benchmark, making it impossible to objectively compare the fault diagnostics capabilities of various AI Agents or to quantify the effectiveness of technology evolution and capability iteration.

Industry urgently needs a unified RCA evaluation standard

As enterprise Agentic Ops enters the stage of large-scale implementation, the lack of an evaluation system has become a key constraint on industry development, and the traditional evaluation paradigm can no longer meet the development demands of artificial intelligence for IT operations:

Traditional evaluation mode completely fails
Root cause analysis is not a simple text processing task. AI Agents must perform real-time indicator queries, log analysis, Tracing Analysis, and change management event assessment, and conduct cross-tool collaborative diagnostics. Traditional evaluation methods that rely on static log fragments and a single label cannot distinguish whether an agent completes full logical reasoning-based diagnostics or merely achieves an accidental hit based on alerting appearances, resulting in significant shortcomings in evaluation effectiveness.
Multi-source observable data is difficult to standardize
RCA evaluation involves multi-source observable signals such as indicators, logs, Tracing Analysis, and system events. These data types are coupled across time and entity dimensions, and failure impact propagates layer by layer along business traces. Taking a database slow query failure as an example, it triggers a chain reaction: increased MySQL query time, increased invoked service latency, upstream service timeout, and frontend 5xx errors. Single-dimension observable data can only render partial symptoms and cannot revert the complete failure propagation logic.
Causal propagation chains easily lead to evaluation misjudgment
The industry commonly confuses abnormal symptoms with failure root causes. Frontend alerting mostly reflects the end of a failure trace, while the real root cause often lies in downstream databases, caches, MSMQ, or the container scheduling layer. If a dataset does not fully depict the causal propagation path and a diagnosis is deemed correct simply by hitting services around the alert, evaluation aliasing is highly likely.
Cross-domain entity identity lacks a unified specification
The same business entity has completely fragmented naming systems across different O&M systems: naming conventions for the same business entity are fragmented across the application performance management, Kubernetes, and cloud resource layers. Evaluation can only rely on character matching or manual subjective judgment, resulting in problems such as unstable scoring, non-reproducible results, and unauditable flows.

In this context, Alibaba Cloud states clearly that building a systematic, standardized RCA Agent evaluation benchmark has evolved from an academic research topic into essential infrastructure for the large-scale implementation of Agentic Ops.

RCA Benchmark Core Definition

RCA Benchmark is not a single-file dataset, but a benchmark suite evaluation system with a complete architecture and closed-loop logic. It consists of three modules: runtime environment, structured sample set, and evaluation protocol.

Runtime Environment: Build a microservice simulation system capable of generating real failure signals, supporting interactive diagnostic queries by AI agents, and completely eliminating the traditional pattern of providing only standard log fragments.
Structured Sample Set: Build a fault sample library with Layer 4 structured ground truth. Each case fully covers four core elements: fault type, normalized root cause entity, causal propagation chain, and key evidence checkpoints.
Evaluation Protocol: Define standardized scoring rules to convert AI agent outputs into quantitative fractions for horizontal comparison. Centered on deterministic rules, this minimizes dependency on Large Language Model (LLM) review and ensures fair and objective scoring.

The project covers all mainstream scenarios, including microservices model failures, database and intermediary failures, Container Orchestration and cloud-native platform failures, cloud resource layer failures, and LLM and agent runtime failures.

RCA Benchmark Core Design Principles and Overall Technical Architecture

RCA Benchmark takes real-world native simulation as its core design concept. It builds a benchmark foundation based on an E-commerce microservice architecture deployed in Kubernetes clusters, containing over 40 business services with call chains up to 7 layers deep. It does not use synthetic data and fully covers typical business dependencies such as synchronous RPC, asynchronous messages, databases, caches, MSMQ, and gateways. With full-domain access to the observability foundation, it supports agents in retrieving seven categories of observation data: indicators, logs, Tracing Analysis, alerting, resource topology, Kubernetes events, and performance profiling. By continuously injecting differentiated background traffic, it replicates production day-night fluctuations, business peaks, and scheduled batch processing payload features to establish a reliable pre- and post-failure comparison baseline.

The project innovatively introduces a four-layer structured ground truth system, abandoning the traditional single root cause label pattern. It completes standardized definitions spanning failure types, normalized entities, causal propagation chains, and key evidence edge zones, with a complementary root cause identification, boundary demarcation, and procedure three-dimension weighted scoring frame that calculates composite scores at 40%, 30%, and 30% weights. Nearly 70% of scores rely on deterministic quantization computation based on failure type topology semantics distance and entity topology distance. Multi-dimensional graded evaluation covers failure semantics matching, topology positioning accuracy, diagnostic evidence, and causal logic completeness, systematically avoiding evaluation bias from random hits. The entire process features transparent rules, reproducible results, and auditable flows.

The platform achieves full-scenario coverage of over 40 failure types across 6 categories at the application layer, intermediary layer, container platform layer, and cloud resource layer through four injection channels: chaos engineering tools, Kubernetes-native O&M, switch configuration, and Alibaba Cloud service APIs. It builds a failure coverage graph across vertical and horizontal dimensions to ensure comprehensive and balanced evaluation scope. To address the industry pain point of fragmented cross-domain entity identities, the platform incorporates a unified entity model (UModel) that assigns cross-domain unique primary keys to all entities, completing multi-domain entity mapping and topology distance calculation through a standardized normalization flow, enabling end-to-end traceability, reproducibility, and auditability.

The system also establishes a four-layer GSTO Quality Gate with multiple admission checks covering structure specifications, signal validity, time windows, and open adaptability settings, strictly filtering invalid samples with failure chain aliasing. Over 200 compliance samples have been accumulated to date, covering all failure type categories and classified into four difficulty levels (L1–L4), with L2 and L3 medium-to-high difficulty scenarios serving as the core evaluation focus.

The project adheres to the principle of open source co-construction. Core capabilities including the evaluation frame, failure directory, scoring protocol, and Quality Gate are fully open source, with co-construction channels open to observability vendors, Agentic Ops developers, and enterprise SRE teams. Reserved non-public test samples and compliance gates prevent data contamination and ensure the fairness and credibility of industry evaluation rankings.

Alibaba Cloud's open source RCA Benchmark establishes a standardized, reproducible, and auditable unified capability ruler for Agentic Ops in the industry, enabling objective benchmarking and quantitative measure of diagnostics capabilities across different agents. Leveraging the tiered difficulty system and full-scenario failure coverage, it supports enterprises in technology selection and business implementation iterations. By open-sourcing core capabilities, it significantly reduces the cost of building in-house evaluation systems. Through dynamic dataset updates, saturation monitoring, and a closed-loop scenario feedback mechanism, the benchmark continuously iterates its capabilities, co-building a long-term evolving, open, and shared O&M intelligent agent industry ecosystem.

Make every failure assessment evidence-based, and every diagnostics capability quantifiable, benchmarkable, and evolvable.

From Black Box to Transparent: Alibaba Cloud Agent Observability and Audit Data Collection in Practice

ObservabilityGuy — Thu, 11 Jun 2026 02:24:01 +0000

This article introduces Alibaba Cloud's LoongSuite solution for comprehensive AI agent observability and audit data collection using extended OpenTelemetry GenAI semantic conventions.

I. Introduction

In 2025, AI agents are moving from the lab to large-scale production. From code assistants used by developers daily to intelligent customer service in enterprise service scenarios, to multi-agent collaboration systems of ever-increasing complexity, AI agents are reshaping software development and business operations at an unprecedented pace.

However, once agents are actually running, a critical problem emerges: the actual runtime behavior of AI agents is difficult to observe, trace, and govern.

A coding agent autonomously and without authorization modifies core configuration files overnight, with no way to know what changed or why. An intelligent customer service agent autonomously issues a "cancel order" instruction, yet the decision logic, tool calling chain, and token resource consumption cannot be reviewed. A multi-agent collaborative job fails midway, and the failure node and root cause are difficult to pinpoint.

These issues point to a common requirement: AI agents need comprehensive observability. Moreover, this observability cannot remain at the shallow statistical dimension of "request success/failure" — it must deeply cover AI agent-specific runtime aspects such as LLM invocation, tool execution, multi-round inference, and memory retrieval.

Based on the OpenTelemetry (OTel) community standard and its in-depth practices in observability fields, Alibaba Cloud has developed a complete data collection solution that covers three types of agent forms. Building on the OTel GenAI semantic conventions, Alibaba Cloud has released the LoongSuite GenAI semantic conventions for observability. This paper will systematically introduce the design concept, technical implementation and use of this scheme.

II. Agent Form Classification and Observability Challenges

The AI agent market is thriving and highly diverse. The runtime models, deployment environments, and use cases of different agent types vary significantly, and their observability and audit needs differ accordingly. We classify mainstream AI agents on the market into three categories:

2.1 Three Major Forms of Agent

2.2 Three Core Challenges
No matter what form is adopted, AI agents will encounter three common problems after large-scale use:

The execution process is black-boxed. The execution process of the agent involves LLM calls, tool execution, multi-round reasoning, and memory retrieval. The traditional Metrics, Log, and Trace methods cannot effectively describe this new computing paradigm. For example, in a round of Agent tasks that contain 10 rounds of ReAct reasoning, the traditional solution can only identify 10 independent HTTP requests and cannot restore a complete hierarchical and orderly decision-making process.
The behavior trajectory is difficult to trace. The agent has high independent operation permissions and can read and write local files, run system commands, and call third-party API operations. Without special audit capabilities, all operations of agents cannot be traced. This poses high risks in enterprise security and compliance control scenarios.
Cost is hard to quantify. Token consumption of large models is the main cost source of agents. Multiple rounds of iterations and tool calls will exponentially increase consumption. Without the ability to fine-tune cost splitting by agent, user, and task, enterprises will not be able to carry out budget control and input-output evaluation.

III. A Differentiated Collection Approach: Adapting to Agents' Native Runtime Forms**

Core design principle: Adapt the data collection capability to the native running mode of the AI Agent instead of forcing the Agent to adapt to the data collection tools.

3.1 Coding Agent: LoongSuite Pilot, a Lightweight Client-Side Data Collector
Coding agents run on the developer's local machine, where all core behaviors — code edits, file creation, terminal command execution — happen in the local environment, completely invisible to traditional server-side agents. To address this, we built LoongSuite Pilot, a client-side data collection platform purpose-built for coding agents.

Core Advantages

One-time deployment, full coverage. Pilot is not a solution exclusive to a single agent, but a unified platform. It currently supports five mainstream coding agents: Claude Code, Codex, Cursor, Qoder, and QoderWork. Developers only need to install it once to automatically collect data from all code assistants in use, with no repeated configuration required.
Silent background execution with zero disruption. Pilot runs as a local daemon process in the background, automatically detecting installed coding agents on the device and deploying capabilities. Developers do not need to modify agent configurations or change usage habits at any point. All behaviors, including LLM invocations, tool execution, and code modifications, are seamlessly recorded.
Resumable collection for stable and reliable data. A built-in breakpoint-resumable collection mechanism handles unstable scenarios such as network fluctuations on local devices, device restarts, and terminal shutdowns. After a process is abnormally interrupted and restarted, no data duplication or data loss occurs, ensuring data integrity.
Flexible collection granularity that balances observability and data security. Different teams have different data security requirements. Pilot supports flexible configuration of collection granularity by agent type. For complete audit needs, detailed info such as message content and tool parameters can be collected. In data-sensitive scenarios, only metadata (model name, token consumption, duration, etc.) is reported, achieving a precise balance between observability requirements and data security.
Plugin architecture, quickly compatible with new agents. Pilot uses a plugin architecture and provides out-of-the-box collection base classes for different agent data formats, such as hook logs, IDE snapshots, SQLite databases, and session files. Integrating a new Coding Agent requires implementing only 2-3 abstract methods, enabling you to quickly keep up with ecosystem iterations. Supported Coding Agents and Coverage

3.2 Personal General-Purpose Assistant: One-Line Command for Full Observability and Audit

Personal general-purpose assistants usually run as standalone services, providing end users with dialogue and task-execution capabilities. For this type of agent, we provide a dedicated plugin that enables full tracing with a single command.

Design philosophy

Take OpenClaw as an example. Although its built-in diagnostics-otel extension can output Metrics and some Trace, it adopts an event-driven architecture. Span is created independently for each event, and there is no parent-child relationship between each other and Trace Context propagation. In essence, it is a group of "standalone data points". The openclaw plug-in of LoongSuite is a complete distributed tracing by design-all Span share the same traceId and are connected together into a call tree through an explicit parent-child relationship.

Span Semantic Model

Each type of span is connected to a complete trace tree by using parent-child relationships. O&M personnel can view the number of large model calls, token consumption, tool call list, time-consuming nodes, and fault information of a single request.

Essential differences from built-in observability

Compared with the built-in observability capabilities of OpenClaw, LoongSuite plug-ins are different in two aspects:

Link integrity. Built-in observability is usually flat and independent, and there is no correlation between events. However, our plug-in is based on the OTel Context propagation mechanism to ensure that ENTRY → AGENT → STEP → LLM / TOOL forms a complete call tree, which can restore the complete picture of a request.

Data richness. Built-in observability often only records basic metrics such as model usage, while our plug-ins fully record fields such as gen_ai.input.messages, gen_ai.output.messages, gen_ai.system.instructions, gen_ai.tool.call.arguments, and gen_ai.tool.call.result to meet the needs of in-depth audit and troubleshooting.

The same plug-in mechanism already covers personal general-purpose assistants such as Hermes Agent and QwenPaw.

3.3 High-and-Low-Code Framework Agent: Zero-Code Instrumentation with the LoongSuite Python Agent
For agent applications built on frameworks such as LangChain, AgentScope, and Dify, the runtime behaves like a traditional Python application. We provide the LoongSuite Python Agent (deeply customized from OpenTelemetry Python Contrib), which achieves zero-code automatic instrumentation with a single command.

Quick start

# 1. Install the LoongSuite Python Agent pip install loongsuite-distro
# 2. Auto-detect and install the required instrumentation libraries
loongsuite-bootstrap
# 3. Start with one command; probes are injected automatically
loongsuite-instrument \
  --traces_exporter otlp \
  --service_name my-agent-app \
  python my_agent_app.py

loongsuite-bootstrap automatically scans for installed frameworks (such as langchain, dashscope, and mcp) in the current environment and installs the corresponding instrumentation packages-developers do not need to manually select and install them.

Framework Coverage

At present, 16 instrumentation libraries have been covered in the LoongSuite Python Agent, covering the mainstream AI agent development framework:

Automatically Recognized Span Types

The probe automatically detects and generates multiple GenAI span types, covering the entire agent lifecycle:

ENTRY: Request entry
AGENT: Agent execution unit
STEP: ReAct reasoning-action iteration step
LLM: LLM invocation, including request parameters, token consumption, and input/output messages
TOOL: tool calling, including tool name, parameter, and result
MCP: MCP protocol invocation
CHAIN: chained invocation orchestration
RETRIEVER: retrieval operations
EMBEDDING: embedding operations
RERANKER: reranking operations
WORKFLOW: workflow orchestration

IV. Observability and Audit Results

After accessing the preceding collection capabilities, users can obtain observability views in the following dimensions. Take Claude Code as an example. If you want to enable Agent Observability, you only need to log in to CloudMonitor 2.0 Console, click the corresponding card in the access center and follow the steps to complete the installation and access with one line of command.

4.1 End-to-End Agent Call Chain View
The complete execution process of the agent is presented in the form of a trace tree, from the user request entry (ENTRY) to the agent decision (AGENT), inference step (STEP), LLM call (LLM), and tool execution (TOOL). The hierarchical relationship is clear at a glance. For complex tasks with multiple rounds of ReAct, you can use Step Span to quickly locate which iteration has a problem, and then go to the LLM or Tool Span in the round to analyze the root cause.

Troubleshooting pattern: When an agent executes a 10-round ReAct process, you can first use Step Span to identify which round of the problem occurred, and then analyze the specific step in the round. This top-down troubleshooting method greatly improves the fault locating efficiency of complex agents.

4.2 Token Usage and Cost Tracking
Based on gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.usage.total_tokens , as well as cost fields extended by Alibaba Cloud (input_cost, output_cost, and total_cost), you can:

Token usage details for a single request
Cost aggregation by agent / user / time
Cache token fields (cache_read.input_tokens, cache_creation.input_tokens) to evaluate cache policy effectiveness

4.3 Session and Multi-Turn Conversation Tracking
Through gen_ai.session.id, gen_ai.turn.id and gen_ai.step.id to build a three-level identification system to achieve:

Full conversation traceability across multiple rounds of conversation

Step-level fine-grained analysis in a single-round dialogue

Session path analysis and user behavior insights

4.4 Tool Call Audit
You can record the tools that are called by the agent, the parameters that are specified, the results that are returned, and the duration. For the Coding Agent, this means that every file read or write and every command execution is documented. For MCP protocol calls, complete request-response auditing is also provided.

Behavior Analysis Dashboard**
**
The top count card divides tool calls into dimensions such as command execution, file reading and writing, search, web browsing, and MCP calls by behavior type, and marks the categories with abnormally high call volume with striking red or orange colors to provide a quick snapshot of the overall behavior composition. The right side displays the number of active sessions and the number of users at the same time, which is convenient for correlating the behavior popularity with the usage scale. The session statistics table below is expanded by session and records the number of calls in each session in each dimension of behavior. This allows you to locate the sessions and users in which high-frequency operations are concentrated.

Tool Call Distribution

The tool invocation distribution page presents the tool usage structure from two perspectives. The pie chart on the left shows the type proportion of all tool calls (such as Read, Write, Bash, TodoWrite, etc.) to help the team understand which tool capabilities the agent relies on most. The pie chart on the right shows the distribution of MCP tool calls independently, revealing which external capabilities are frequently called in cross-system integration. The trend comparison chart below shows the changes in the number of calls for each tool type in a timeline, making it easy to identify phased changes in call patterns-for example, a surge in Bash calls on a certain day may indicate batch script tasks or abnormal behavior.

Security Audit Overview

The Overview page compresses the security situation of AI agents into a screen-readable risk snapshot based on the multi-dimensional high-risk operation count within a specified time window. The funnel on the left side gradually converges from full sessions to sessions with security risks. This visually shows the proportion of risk surfaces. On the right side, metrics such as high-risk command execution, outbound web requests, outbound command-line requests, sensitive file access, and prompt injection are displayed side by side. With the comparison data, the security team can quickly determine whether the current risk level is abnormal without in-depth details.

What is particularly noteworthy is the count of high-risk operations after the prompt injection event. Ordinary high-risk operations may originate from the reasonable requirements of the task itself, while high-risk behaviors triggered by injection are strong threat signals-this means that the injected malicious instructions have driven the Agent to execute. Even if there is a false positive, such signals should trigger a manual review at the highest level, rather than waiting for further confirmation. Therefore, the “number of tool-calling sessions following prompt injection” is the highest-confidence Indicator of Compromise (IoC) in the entire overview. The priority of 3 such sessions is often higher than that of hundreds of ordinary high-risk commands.

High-Risk Session Tracing

Two-stage drill-down capability is provided below. The upper layer is a high-risk session risk score table, which aggregates the risk counts of each dimension (injection hits, high-risk operations, sensitive file accesses, and outbound information) by session, and automatically sorts the comprehensive risk score to present the sessions that require the most manual intervention. The security team does not need to screen logs one by one. Instead, the security team directly starts tracing from the session with the highest risk, greatly reducing the time window from discovery to response.

The lower layer is a high-risk event summary table, which drills risk down to individual event granularity-specific time, user, session, event type, tool name involved, threat type, and complete context content, providing security analysts with the original evidence required for final characterization.

V. Deep Extensions Based on the OTel GenAI Semantic Conventions

The data capabilities of the observability system of Alibaba Cloud AI Agent are built based on the self-developed LoongSuite GenAI Observability Semantic conventions. This specification is based on the OTel GenAI standard in the community and fills the semantic gaps in real business scenarios.

5.1 Why Extend Beyond Community Standards
As early as the beginning of 2024, OpenTelemetry started driving GenAI semantics specification development, aiming to establish a unified observability data language. Community standards have laid an important foundation:

gen_ai.operation.name: Standardized operation types (chat, embeddings, execute_tool, etc.)
gen_ai.span.kind: Differentiates span types such as LLM, CHAIN, AGENT, TOOL, and RETRIEVER
gen_ai.request.model / gen_ai.response.model: Model identity
gen_ai.usage.input_tokens / output_tokens / total_tokens: Token usage
gen_ai.input.messages / gen_ai.output.messages: Input and output messages
gen_ai.response.finish_reasons: Model stop reason

However, community standards inherently need to balance broad applicability with long-term stability, resulting in a relatively cautious pace of evolution. The current OTel GenAI semantic conventions is still in Development status, and many new concepts and scenarios are still being absorbed and converging.

In practice at Alibaba and Ant Group, we encountered many more complex and granular real-world scenarios. For example, a seemingly simple scenario of "ordering milk tea with Qwen" actually involves cross-domain coordination among multiple business systems, including Qwen Agent, Flash Sale Agent, Amap Agent, and Alipay Agent. These scenarios place higher demands on semantic expressiveness.

To this end, based on the OTel GenAI community standard and drawing from extensive internal hands-on experience, we released the LoongSuite GenAI Observability Semantic conventions. In 2026, the specification was officially open-sourced as a vendor extension standard for OTel GenAI, with plans to gradually contribute optimization capabilities upstream to the community.

5.2 Selected Core Extensions
Extension 1: Entry Span and Step Span — Making Complex Agent Call Chains Readable

Problem background: When an agent executes a long-running job, a single trace may contain hundreds or even thousands of spans. The native standard cannot distinguish business levels, making call chains cluttered and difficult to analyze.

Semantic Modeling:

Entry Span (gen_ai.span.kind = ENTRY ): Created at the entrance of the agent call, used to restore the original input and output of the model and the user to form the dialogue history. Ensure that when processing downstream tasks, the data is not polluted by System Prompt or framework Prompt, and the most original customer request can be obtained.
Step Span (gen_ai.operation.name = react ): represents the hierarchical expression of Agent in each ReAct process. Each ReAct completes the cycle of "reflection → tool invocation → model invocation", identifying the turn by gen_ai.react.round. The round-by-round span structure makes the trajectory of each loop clear at a glance. This semantic conventions has been implemented in multiple scenarios such as OpenClaw, QwenPaw, and Hermes Agent.

Extension 2: Skill Semantics — Making Business Function Domains Observable

Background: In scenarios such as e-commerce shopping assistants, commands are routed to the corresponding Skill after the agent understands the intent. Existing semantic conventions lack an abstraction of the business function aggregation layer of Skill.

Semantic Modeling: gen_ai.skill.* attribute family is added:

At the current stage, these attributes are attached to the execute_tool Span and quickly landed. At the same time, we have implemented an independent invoke_skill Span scheme and submitted a proposal to the OTel community (#3540).

Downstream value: Observability Platform can be aggregated and analyzed by functional domain to quickly identify "which Skill has the highest error rate", compare "whether the latency of the new version of Skill is degraded after it is launched", and measure "the proportion of Skill execution time spent on LLM calls".

5.3 Engineering Implementation: GenAI Utils
The value of semantic conventions lies not only in documents, but also in engineering implementation. We implemented GenAI Utils in the probe as an engineering capability layer for the LoongSuite SemConv:

Data extraction only at the instrumentation layer: Each framework instrumentation library intercepts framework calls by using hooks or Monkey-Patch, and fills data into the corresponding Invocation data object.
GenAI Utils unified telemetry output: All span creation, attribute mounting, metrics recording, event sending, and context management are completed by the ExtendedTelemetryHandler.
Only one specification update: When LoongSuite SemConv adds new fields or adjusts the structure, you only need to modify GenAI Utils. All downstream instrumentation libraries automatically take effect.

Supported Invocation types include LLMInvocation, InvokeAgentInvocation, CreateAgentInvocation, ExecuteToolInvocation, EmbeddingInvocation, RetrieveInvocation, RerankInvocation, and MemoryInvocation, covering the entire lifecycle of GenAI.

GenAI Utils has versions of Python, Node.js, and Go, and the Java version will be released soon. Among them, Python and Node.js versions have been open-sourced, and the rest will be open source one after another.

VI. Summary

The Alibaba Cloud Agent observability and audit solution is applicable to the following scenarios:

The popularity of AI agents has greatly improved production and office efficiency, and also put forward new requirements for observability, auditability, and governance capabilities. Different from traditional microservices and web applications, AI Agent integrates new operation modes such as LLM calls, tool execution, and multi-turn reasoning. It must support exclusive data collection and semantic standards.

The Alibaba Cloud LoongSuite solution provides full coverage for the following types of mainstream agents:

LoongSuite Pilot eliminates blackboxes for locally running coding agents such as Claude Code, Cursor, Codex, Qoder, and QoderWork.
Dedicated plug-ins (OpenClaw, Hermes Agent, QwenPaw) give personal general-purpose assistants full tracing capabilities.
The LoongSuite Python Agent, which is open source and uses 16 framework instrumentation libraries, allows agent applications developed based on frameworks such as LangChain, AgentScope, Dify, and MCP to implement zero-code access.

More importantly, the LoongSuite GenAI Observability Semantic conventions, which is based on the OTel GenAI Semantic conventions, is open source. It uses key semantic extensions such as Entry, Step Span, and Skill semantics to fill the semantic gaps of community standards in real business scenarios. With the engineering package of GenAI Utils, this ensures unified standard implementation and efficient iteration.

The ultimate goal of a unified semantic conventions is not to produce a single document, but to enable all users and vendors who use the specification to see, analyze, govern, and evolve the rapidly growing GenAI applications.

Related links:

Cloud Monitor 2.0 console: https://cmsnext.console.aliyun.com/
AgentLoop console: https://agentloop.console.aliyun.com/
Semantic conventions: https://github.com/alibaba/loongsuite-semantic-conventions-genai/
Python Agent: https://github.com/alibaba/loongsuite-python-agent

Beyond 'Demo-Grade' Architecture: Building a Highly Available Production Foundation for Dify with SAE SLS

ObservabilityGuy — Fri, 05 Jun 2026 03:10:52 +0000

This article introduces Alibaba Cloud SAE, a serverless platform that simplifies application modernization and accelerates AI deployment with zero node management.

Introduction
When facing complex microservice operations and volatile AI traffic patterns, building an elastic, maintenance-free "compute foundation" is also crucial.This article expands the scope from data architecture to full-stack infrastructure, introducing the ultimate production-grade solution built on Alibaba Cloud SAE × SLS.

With the explosive growth of LLM-powered applications, Dify—with its powerful workflow orchestration and user-friendly visual interface—is becoming the go-to platform for building enterprise AI applications. However, when applications move from local demos to large-scale production, developers often hit two "hidden" challenges: skyrocketing operational complexity and data architecture performance bottlenecks.

This article provides a deep analysis of these architectural bottlenecks and introduces the joint solution built on Alibaba Cloud SAE (Serverless App Engine) and SLS (Simple Log Service)]. Through the dual engines of "fully managed compute" and "storage-compute separation," we build a highly elastic, cost-efficient Dify production environment with deep data insights.

1.Current State and Challenges: Architectural Bottlenecks in Scaling Dify

During the single-machine demo phase, deploying with Docker Compose and the default PostgreSQL storage is perfectly adequate. But once you enter production, these two pieces of infrastructure are often the first to become performance and scalability bottlenecks.

▍Operational Complexity
Dify is a microservice architecture composed of multiple components: API service, Worker, Web frontend, KV cache, relational database, and vector database. In production, this architecture poses significant operational challenges:

· Lack of resource elasticity: AI applications typically exhibit pronounced traffic peaks and valleys. With self-managed Kubernetes or ECS clusters, scaling responses lag behind demand—users queue during peaks, while massive resource waste occurs during off-peak hours, driving up costs.

· High maintenance costs: Ensuring high availability, configuring load balancing, handling node failures, and performing blue-green or canary deployments—this foundational infrastructure work carries a high technical bar and consumes significant engineering effort that should be spent on business innovation.

· Performance bottlenecks: The default deployment provides limited QPS capacity, making it difficult to support high-concurrency scenarios—especially under inference-intensive workloads, where it easily becomes a system bottleneck.

▍Database Capacity Explosion
By default, Dify stores all data—including business metadata and runtime logs—in PostgreSQL. As business volume grows, the mismatch between data characteristics and the storage engine becomes increasingly apparent:

• Logs "bloat" the database: Every workflow node execution generates a complete record of inputs, outputs, prompts, reasoning processes, and token statistics. In high-concurrency production scenarios, this data consumes the vast majority of database resources, causing tablespace to expand rapidly.

• Core business degradation: High-frequency, high-throughput log writes consume database connection pools and I/O resources, severely interfering with core business operations (such as creating applications, knowledge base retrieval, and conversation context management), leading to response delays, timeouts, and even service unavailability.

2.Synergistic Empowerment: SAE and SLS Core Advantages

To address these bottlenecks, SAE and SLS work in tandem—SAE focuses on elastic compute scheduling, while SLS specializes in massive log storage—together building a high-performance, highly available runtime foundation for Dify.

▍SAE: A Fully Managed, Elastically Scalable Runtime for Dify
SAE handles more than just orchestrating Dify's core microservices (API, Worker, Sandbox). Through one-click templates, it integrates the complete cloud ecosystem required to run Dify.

• One-click full-stack delivery: Developers no longer need to manually build complex environments. Using pre-built templates, you can deploy a complete microservice cluster with a single click, automatically creating and integrating SLS (workflow log storage), Tablestore (vector storage), Redis (caching), and RDS for PostgreSQL (metadata storage)—no need to purchase and configure each service individually, delivering a "production-ready out of the box" experience.

• Enterprise-grade high availability: Instances are automatically distributed across multiple availability zones, combined with health checks and self-healing mechanisms to prevent single points of failure. Canary deployments ensure smooth, seamless traffic shifts during frequent workflow iterations.

• Sub-second compute elasticity: A perfect fit for the "tidal" characteristics of AI workloads. SAE supports auto-scaling based on CPU/memory utilization or QPS metrics. During inference peaks, Worker instances spin up in seconds to absorb pressure; during off-peak periods, idle resources are automatically released, keeping compute costs strictly within the "actual usage" range.

• Deep performance tuning: SAE has applied end-to-end, code-and-architecture-level tuning to Dify—not only patching Redis cluster compatibility and slow SQL issues at the infrastructure layer, but also fine-tuning runtime parameters and aligning resource specifications. This full-stack optimization drives a 50x throughput leap from 10 QPS to 500 QPS, ensuring silky-smooth AI responses.

▍SLS: A "Storage-Compute Separation" Solution for Massive Data
SLS is not simply a database replacement—it is cloud-native infrastructure purpose-built for log scenarios. Compared to PostgreSQL, SLS delivers architectural upgrades across four dimensions in the Dify context:

• Extreme storage elasticity: Unlike databases that require resource provisioning based on peak loads, SLS as a SaaS service natively supports sub-second elastic scaling. Whether it's a late-night trough or a sudden inference spike, it adapts automatically—no need to worry about sharding or capacity limits.

• Architectural decoupling and load isolation: By leveraging append-only write patterns, SLS avoids the random I/O and lock contention common in databases, easily supporting 10,000+ TPS throughput. By completely offloading the log workload to the cloud, it ensures that massive log writes do not affect Dify's core business response times.

• Tiered storage for cost-efficient retention: Powered by high compression ratios, hot data is analyzed in real time while cold data automatically sinks to archive storage. This meets long-term audit and retrospective needs at costs far below database SSD pricing.

• Out-of-the-box business insights: The built-in OLAP analysis engine supports real-time SQL queries, visual dashboards, and alert monitoring, helping developers transform dormant log data into actionable business insights.

3.Effortless Deployment: Define a Production-Grade Foundation in 1 Minute

The SAE App Center includes a deeply optimized Dify production template. With simple parameter configuration, you can deploy a highly available runtime environment in a single click—no more tedious YAML writing and environment debugging.

Step 1: Select a deployment template
Log on to the SAE console, go to the App Center, and select "Dify Community Edition - Serverless Deployment."

Step 2: Configure parameters and select specifications
Three templates are currently available: Dify High-Performance Edition, Dify High-Availability Edition, and Dify Test Edition.

For high-concurrency production scenarios, we recommend the Dify High-Performance Edition, which includes deep optimizations specifically for the api image and plugin-daemon image, resulting in higher runtime efficiency. Configuration is streamlined—simply fill in the passwords for each cloud service and select the VPC and vSwitch. The system then provides a total estimated price for the selected cloud resources, ensuring cost transparency.

Step 3: Submit and access the service
Click Submit, and the system automatically completes the deployment of core services and cloud resource associations.

After deployment, enter the service address provided by the console—${EXTERNAL-IP}:${PORT}—directly in your browser to begin your Dify application orchestration journey.

Note: After Dify starts and is running, the SLS plugin automatically creates the relevant logstores and index configurations. No manual intervention is required—simply navigate to the corresponding project in the SLS console to query and analyze workflow logs in real time.

4.50x Performance Leap: SAE's Journey from 10 QPS to 500 QPS

Dify Community Edition's default configuration supports only 10 QPS, but that's just the starting point. Scaling from "getting started" to 500 QPS production capacity isn't a matter of simply throwing more server resources at the problem—it's a step-by-step "boss fight." Every time you try to increase throughput, you hit a new invisible ceiling—from basic parameter limits to deep architectural bottlenecks. The SAE team used full-stack load testing to map out and conquer the two core checkpoints on this progression, making high-performance deployment a well-charted path.

▍Bottleneck 1: Breaking the 10 QPS Limit—Coordinated Tuning of Component Concurrency and Database Connections
1.Why does the default configuration cap at 10 QPS?
Dify Community Edition's default configuration is designed for quick developer tryout, not large-scale production. The default parameters for its core component dify-api are extremely conservative:

SERVER_WORKER_AMOUNT (worker processes): 1
SERVER_WORKER_CONNECTIONS (max connections per process): 10

These two parameters directly cap the throughput of a single node. But in production, you cannot simply "multiply by ten"—increasing application-layer concurrency immediately triggers a chain reaction in downstream databases.

2.The "connection pool" domino effect
As QPS grows, components like dify-api and dify-plugin-daemon open massive numbers of connections to PostgreSQL. Without end-to-end parameter coordination, the system easily collapses:

• Connection exhaustion: PostgreSQL has a finite total connection limit. Blindly increasing component concurrency drains database connections, causing subsequent requests to fail outright.

• Connection contention between components: SQLAlchemy's connection pool uses a "lazy loading" mechanism, and idle connections are not released until they expire. If misconfigured, non-critical components can hoard large numbers of idle connections while critical components starve for resources during peak traffic.

Solution: A battle-tested "production-grade configuration matrix"
To prevent users from falling into a cumbersome parameter trial-and-error cycle, the SAE team conducted multiple rounds of full-stack load testing in real production environments. They identified the production-grade configuration matrix mapping API concurrency, database connection pool sizes, and component resource specifications across different traffic tiers. Users don't need to worry about parameter calculations—simply select the specification tier matching your estimated traffic to ensure every unit of compute translates into actual business throughput.

Note: The load testing scenarios do not include the code execution (Code Sandbox) path. Please evaluate and adjust the specifications and quantity of the dify-sandbox component based on the complexity of code execution in your actual business.

Configuration reference: https://help.aliyun.com/zh/sae/dify-performance-optimization

▍Bottleneck 2: From 200 QPS to 500 QPS — Redis Single-Point Bottleneck and Read-Write Separation
1.Integrating ARMS tracing to identify performance bottlenecks
After optimizing database connections and stabilizing QPS at 200, the system throughput could not be pushed further. To locate the bottleneck, the SAE team used ARMS application monitoring deeply integrated into the SAE platform to perform trace analysis on the dify-plugin-daemon component—on the SAE console's application details page, click "Application Monitoring" to view the slowest call chains.

Trace data revealed that downstream Redis SET/DEL operations were failing frequently. The SAE team attempted to vertically scale the Redis instance to the maximum specification (64 cores), but the effect was minimal: the QPS ceiling did not improve, indicating that the bottleneck was not in capacity, but in the single-point architecture itself.

2.dify-plugin-daemon's high-frequency Redis reads and writes causing single-point congestion
Code analysis revealed that this was a conflict between Dify's business logic and Redis's single-point architecture:

• dify-plugin-daemon generates a new Session ID for every data pipeline request and writes it to Redis. This session data is then read and verified on every subsequent request. This creates a pattern of high-frequency, small-payload read-write operations concentrated on a single key space.

• In the default architecture, all session read-write requests are concentrated on a single Redis node. Under 200+ QPS high-concurrency pressure, the single node becomes a throughput bottleneck—not due to insufficient memory, but because the network I/O and single-threaded command processing of a standalone Redis instance cannot handle the concurrent connection load.

Solution: Cluster transformation for read-write separation
To break through the single-machine architecture limitation, the SAE team went deep into the component internals and performed cluster adaptation for dify-plugin-daemon:

• Cluster protocol support: To address the native component's lack of Redis Cluster support, the SAE team modified the underlying code to fully support the Redis Cluster protocol, including hash-slot-aware key routing and cluster node auto-discovery.

• Read-write separation: Through architectural upgrade, the massive requests originally concentrated on a single machine were distributed across the cluster. The cluster's multi-node characteristics enable load distribution and read-write separation.

This transformation completely eliminated the single-point bottleneck, successfully supporting a smooth throughput increase from 200 QPS to 500 QPS.

5.Unlocking Full-Stack Data Value: SLS Transforms "Black Box Operations" into "Deep Insights"

Once Dify is live, how do you assess model costs and performance? How do you analyze business trends? Powered by SLS's robust OLAP analysis engine, you can perform deep mining of Dify's workflow logs without pre-defining table schemas, building comprehensive dashboards covering both technical and business metrics.

▍Infrastructure Perspective: LLM Cost and Performance Transparency
For Dify's LLM nodes, the process_data field in workflow_node_execution logs contains detailed model invocation data, enabling sub-second multi-dimensional analysis of model usage.

Scenario A: Token Consumption and Cost Auditing
Real-time monitoring of token consumption trends is key to controlling AI costs. You can track input tokens (prompt_tokens), output tokens (completion_tokens), and total tokens over time, precisely identifying anomalous traffic.

Sample SQL:

node_type:llm | select
  sum(
json_extract_long(process_data, '$.usage.prompt_tokens')
) prompt_tokens,
sum("process_data.usage.completion_tokens") completion_tokens,
sum("process_data.usage.total_tokens") total_tokens,
date_trunc('minute', __time__) t
group by
  t
order by
  t
limit
  all

Note: Fields within JSON can be extracted directly in SQL using json_extract_xxx functions, such as json_extract_long(process_data, '$.usage.prompt_tokens'). For frequently used fields, we recommend creating additional JSON sub-indexes so you can reference the column name directly in SQL, such as "process_data.usage.completion_tokens", for more efficient statistical analysis.

Scenario B: Time-to-First-Token (TTFT) Percentile Analysis
LLM response speed directly impacts user experience. By analyzing the P50, P90, and P99 percentiles of time_to_first_token, you can objectively evaluate model response stability under different loads, providing data support for model routing or inference acceleration decisions.

Sample SQL:

node_type:llm| select
  date_format(__time__-__time__ % 60, '%m-%d %H:%i') as time,
   approx_percentile("process_data.usage.time_to_first_token", 0.25) as Latency_p25,
  approx_percentile("process_data.usage.time_to_first_token", 0.50) as Latency_p50,
  approx_percentile("process_data.usage.time_to_first_token", 0.75) as Latency_p75,
  approx_percentile("process_data.usage.time_to_first_token", 0.99) as Latency_p99,
  min("process_data.usage.time_to_first_token") as Latency_min
group by
  time
order by
  time
limit
  all

▍Business Operations Perspective: User Intent and Conversion Insights
Beyond low-level model metrics, SLS can help you understand business logic at a deeper level. Using an "e-commerce AI customer service assistant" Dify application as an example, you can use SQL to dissect workflow node inputs and outputs to support operational decisions.

Scenario A: User Intent Distribution Trends
By analyzing the output of the "intent recognition" node in the workflow, you can quantify the most frequent user inquiry categories (e.g., returns/exchanges, shipping inquiries, coupons), and observe how these demands change over time—guiding knowledge base optimization efforts.

Sample SQL:

* and title: User intent recognition | select
  json_extract(outputs, '$.text') as "user intent",
  count(1) as pv
group by
  "user intent"

Scenario B: Anomaly Diagnosis and Funnel Analysis
By tracking error rates for specific nodes or analyzing the downstream flow of specific intents, you can build funnel charts to quickly identify nodes causing user drop-off. For example, analyzing the "empty result" rate of the "product search" node can indicate whether the product knowledge base needs expansion.

You can use funnel charts to analyze and observe which intermediate workflow nodes have a high failure rate.

Sample SQL:

status:succeeded | select
title,
count(distinct workflow_run_id) cnt
group by
  title
order by
  cnt desc

6.Conclusion: Let AI Applications Focus on What Matters

From "functional" to "production-ready," Dify's journey to production-grade deployment requires solid infrastructure support. The SAE × SLS joint solution is not just a simple combination of two cloud products—it delivers a full-stack Serverless architectural transformation for Dify through deep integration of "compute management" and "storage decoupling":

• Full-stack elasticity: The compute layer scales in seconds with traffic, the storage layer handles burst throughput effortlessly—a perfect match for the tidal characteristics of AI workloads.

• Structural cost reduction: Eliminates idle resource waste completely. Replaces expensive database expansion with low-cost tiered storage, maximizing ROI.

• Extreme stability: A fully managed, maintenance-free foundation combined with physical I/O isolation completely eliminates single-point-of-failure risks and database performance black holes.

• Deep insights: Breaks the "black box" between infrastructure monitoring and business data analytics, using token cost and user intent data to fuel business evolution.

With this solution jointly released by SAE and SLS, Dify developers no longer need to worry about underlying resources and architecture. A single, simple configuration gives you a highly available, high-performance, cost-efficient AI application environment—allowing you to truly focus on business innovation and prompt tuning.

Get started now: Log on to the Alibaba Cloud SAE console[1], go to the App Center, search for the Dify template, select the Dify High-Performance Edition, and start your one-click managed deployment journey.

▍https://alidocs.dingtalk.com/i/nodes/gvNG4YZ7Jnxop15OC9ZogOKgW2LD0oRE?utm_scene=team_space
Alibaba Cloud Serverless App Engine (SAE) is a one-stop containerized application hosting platform built for the AI era, with the core philosophy of "supporting traditional applications and accelerating AI innovation." It simplifies operations, ensures stability, reduces costs by up to 75% through idle resource optimization, and enhances operational efficiency through an AI-powered assistant.

For AI workloads, SAE integrates mainstream frameworks like Dify, supporting one-click deployment and elastic scaling. In the Dify scenario, it achieves a 50x performance improvement and over 30% cost optimization.

Product Strengths
With eight years of technical refinement, SAE was named a Global Leader in the 2025 Gartner Magic Quadrant for Cloud-Native Platforms—ranked #1 in Asia—helping enterprises achieve zero node management and focus purely on business innovation. SAE serves as both a "hosting platform" for traditional application modernization and an "acceleration engine" for large-scale AI application deployment.

1.Traditional Application Operations: The "Simplify, Stabilize, Save" Approach

• Simplify: Zero operational overhead — focus on business innovation

• Stabilize: Enterprise-grade high availability with built-in comprehensive protection

• Save: Extreme elasticity that brings costs down to measurable levels

2.Accelerating AI Innovation: From Rapid Exploration to Efficient Deployment

• Rapid exploration: Built-in templates for Dify, RAGFlow, OpenManus, and other popular AI applications — ready out of the box, with POC up and running in minutes;

• Reliable deployment: Production-grade AI runtime with performance optimizations (e.g., 50x performance boost for Dify), seamless upgrades, and multi-version management for enterprise-grade reliable delivery;

• Easy integration: Deep integration with gateways, ARMS, metering, and auditing capabilities to accelerate the intelligent transformation of traditional applications.

Who is it for?
✅ Startups: No dedicated ops team, need to launch quickly
✅ SMBs: Looking to cut costs and embrace cloud-native
✅ Large enterprises: Requiring enterprise-grade stability and compliance
✅ Global businesses: Needing China + worldwide deployment
✅ AI innovation teams: Looking to rapidly deploy AI applications

Learn more
Product page: https://www.alibabacloud.com/product/severless-application-engine

Related Links:
[1] Alibaba Cloud SAE console
https://saenext.console.aliyun.com/overview?accounttraceid=db100a4af9c7405e88dcfb89e81c5281ibby

DEV Community: ObservabilityGuy

Say Goodbye to Complex Onboarding: AI Agent Skills Drive Cloud Monitor's Observability Integration

Background

CMS CLI Overview

Application Integration Capabilities

Achieving Intelligent Integration via AI Agent Skills

From Individual Productivity to Organization Capability: AI Coding Measure Practices of LoongSuite-Pilot and SLS

Introduction: Coding Has Become Faster, But Why Has the Organization Not Kept Up?

Data Ingestion Layer: from Collection to SLS Implementation

Measure Layer: from a Public CTE to an Actionable AI Coding Agent Measure Dashboard

Conclusion

Related Links

Kernel-Level X-Ray Vision: CloudMonitor 2.0 Achieves Full-Stack Observability with Zero Code Changes

Background

Three Pillars: Application Monitoring, Network Monitoring, and Log Enhancement

Protocol Panorama: From HTTP to CUDA, Catching Everything in One Net

Language depth Integration: Not limited to the Network Layer

The Complete Lifecycle of a Span (Data Pipeline and Architecture)

GPU/CUDA Tracking: Covering Kernel Launches and VARM Operations

Network Layer Observation: Traffic, Association, and Quality

Use OBI (OpenTelemetry Non-intrusive Monitoring) in CloudMonitor 2.0

Electron Monitoring: Making Desktop Agent Monitoring Accessible

Introduction: The "Monitoring Blind Spot" on the Desktop Side

Why is Electron Monitoring Difficult?

Breakthrough Idea: One init(), Full-end Observability

Six Core Capabilities to Eliminate Monitoring Blind Spots One by One

Compared with Competitors, What Are the Advantages?

Quick Intergation: Simpler than You Think

Performance and Overhead: Lightweight Enough to Be Ignored

Scenarios

Conclusion

Can the Log Be Changed? SLS Logstore Adds Native Support for Log Updates and Deletes

Introduction

I. Foreword

II. Design Tradeoffs

III. Capability Forms

IV. Typical Scenarios

V. Final Thoughts

Enabling Boundless Intelligent Collaboration: UModel Now Open Source, Launching the Universal Semantics Standard Initiative

UModel: Object Graph Semantic Runtime for Enterprise AI

From UModel to Universal Semantic Standards: Something No Single Enterprise Can Accomplish Alone

Conclusion

Alibaba Cloud Launched a Global Intelligent O&M Platform from "Passive Fire Fighting" to "Active Autonomy"

From API to AI Agent: Alibaba Cloud CloudMonitor Command-line Interface (CLI) + Agent Skill in Practice

30 Second Overview

Why You Need CLI + Agent Skill

CLI + Skill Solutions

CLI + Skill Workflow

Practical Scenario 1 (Integration Center): Integrate Container Service for Kubernetes (ACK) Clusters into CloudMonitor

Practical Scenario 2 (Alerting Center): Intelligent Alert Rule Management

Practical Scenario 3 (Prometheus service): Prometheus Instance Management and Data queries

Practical Scenario 4 (Application Performance Monitoring (APM)): Application Monitoring/AI Observability Onboarding

Practical scenario 5 (Data query): metadata, Prometheus Query Language (PromQL), and CloudMonitor Basic metric queries

Summary

Appendix—CMS CLI Command Tree

Enabling Boundless Intelligent Collaboration: UModel Now Open Source, Launching the Universal Semantics Standard Initiative

1.UModel: Object Graph Semantic Runtime for Enterprise AI

2.Technical Advantages of UModel

3.Umodel Evolution Route Planning

4.Administration Models and Participation Paths

5.Quick Start

From UModel to Universal Semantic Standards: Something No Single Enterprise Can Accomplish Alone

6.Conclusion

Alibaba Cloud Releases RCA Benchmark, the Industry's First Open Source Root Cause Analysis Benchmark System for Agentic Ops

From Black Box to Transparent: Alibaba Cloud Agent Observability and Audit Data Collection in Practice

I. Introduction

II. Agent Form Classification and Observability Challenges

III. A Differentiated Collection Approach: Adapting to Agents' Native Runtime Forms**

IV. Observability and Audit Results

V. Deep Extensions Based on the OTel GenAI Semantic Conventions

VI. Summary

Beyond 'Demo-Grade' Architecture: Building a Highly Available Production Foundation for Dify with SAE SLS

1.Current State and Challenges: Architectural Bottlenecks in Scaling Dify

2.Synergistic Empowerment: SAE and SLS Core Advantages

3.Effortless Deployment: Define a Production-Grade Foundation in 1 Minute

4.50x Performance Leap: SAE's Journey from 10 QPS to 500 QPS

5.Unlocking Full-Stack Data Value: SLS Transforms "Black Box Operations" into "Deep Insights"

6.Conclusion: Let AI Applications Focus on What Matters

Breakthrough Idea: One `init()`, Full-end Observability