DEV Community: ObservabilityGuy

Add Enterprise Memory to OpenClaw, and Your Agent Finally Doesn’t Have to Ask Again

ObservabilityGuy — Tue, 26 May 2026 03:12:47 +0000

This article introduces AgentLoop MemoryStore, a fully managed, enterprise-grade memory solution designed to give AI Agents long-term, reliable memory for production environments.

Presumably every AI developer has experienced such a scenario: your intelligent Agent is finally online. Demo ran smoothly, the internal review passed smoothly, and the boss nodded his approval. After two months of hard work, the team finally pushed it into the production environment. In the first week, user feedback was acceptable. But by the second week, you receive a user message like this: "The last time I explicitly said I wanted to return it, why is your robot still asking me if I want to exchange it?" You go through the conversation log, and what the user said is true-in the last round of dialogue, the intention to return was very clear. However, Agent has no impression. Every conversation is like meeting for the first time. You suddenly realize: Agent online is only the starting point, the real key is that it must "remember". And the pain behind this is far deeper than imagined.

The First Layer of Pain: Users Would Not Like to Say It Again
This is the most direct experience of harm, but also the most silent reason for the loss of users. Users don't care about your technical architecture or which big model you use. All they know is that what they said yesterday will be repeated today. In the customer service scene, the user has already explained the order problem, the receiving address and the return request, but he has to repeat it from the beginning when he enters the line again. The experience collapses instantly and the customer complaint rate rises sharply. In the sales scene, the customer made it clear that "the budget has not been approved" before, and Agent still repeatedly pushes the quotation scheme, which will only make the customer feel that the assistant is not listening at all. In the learning scene, the next day, the system still repeatedly questions as weak items, which will only make people feel that the product is perfunctory.

Users will not complain about "your memory system is not working", they will only lose it silently, or be prepared before the next use-it can't remember what I said anyway.

The Second Layer of Pain: On the Road to Self-Study, You Have to Step on the Pits Yourself
After noticing the problem, many teams chose to develop their own memory system, only to find that the road was far more difficult than expected. Originally three weeks to complete the memory function, eventually evolved into three months of the underlying infrastructure reconstruction.

● Easy to store but difficult to recall: It is not complicated to store the dialogue history in the vector database. The difficulty is to accurately recall the "most relevant information" in the next round, rather than bringing back a bunch of invalid noise. If the retrieval quality is not up to standard, the memory will be useless, recalling five pieces of information and four pieces of interference, but will bias the model judgment.

● Only increase but not decrease, memory confusion: users prefer concise answers last month, and this month they want to explain in more detail. If the system only adds but not updates, the two contradictory information coexist, and the more dirty data they use, the more inconsistent judgments.

● Context stacking and effect reversal: Some people directly put all the history into the Prompt, which seems simple, but leads to double the token cost and slow response. The model filters valid content from redundant information, and the accuracy does not increase but decreases. Long context doesn't equal good memory, and many times it's just more expensive noise.

● Demo is smooth and production is unstable: The memory of a single machine performs well in the testing phase. In the first production phase, problems occur frequently, such as the memory of multi-instance deployment does not communicate with each other, the memory of instance destruction is lost, and the memory extraction of high concurrency slows down the main link...

The Third Layer of Pain: The Function Is Done, but I Dare Not Go Online the Main Link
This is the most hidden and most realistic pain point. The memory function can be realized technically, but after landing, the problem ensues: who will maintain the vector database? How do I troubleshoot and locate exceptions? User historical memory involves privacy. How can data isolation be ensured? Compliance requires that the memory can be traced and deleted. Can the existing scheme be supported? Will the memory assembly line drag down the entire service if the traffic surges tenfold? Before these questions are clearly answered, any prudent technical leader dares to connect the core agent to the primary link. Memory is not unable to do it, but after it is done, no one dares to be really responsible. As a result, a large number of agents in the team are in an awkward position: the functions are already available, the project is not ready, and the business is slow to deliver.

In the past few years, memory ability has almost become the most crowded track in agent infrastructure. Simply storing conversations, enabling vector retrieval, and recording user preferences are no longer scarce capabilities. What is really scarce is an enterprise-level memory system that allows enterprises to quickly access, fit business scenarios, and run stably in the production environment. This is the core problem AgentLoop MemoryStore want to solve. As a fully managed enterprise-level memory management agent, AgentLoop MemoryStore has three advantages: out-of-the-box, flexible customization, and serverless O&M-free. It is equipped with core capabilities such as multi-dimensional memory retrieval, intelligent memory update, asynchronous pipeline architecture, and hierarchical precision retrieval. It no longer asks "memory weight is not important"-the answer you already know. What it needs to solve is: why the enterprise has been slow to put the core agent online, and how this key point is completely broken.

For agents, the value of memory goes far beyond "preserving historical conversations." It determines whether the agent can upgrade from a one-time question and answer tool to a long-term collaboration partner that continuously understands users, reuses context, and deposits business experience. Without memory, each round of Agent dialogue is like a first meeting. With reliable memory, Agent can truly understand "who you are, what happened, and how to continue judgment".

For enterprises, memory is never an additional function, but a watershed of whether Agent can really be used. Does the customer service robot remember the user's last work order? Does the sales assistant remember the customer's decision-making progress and historical objections? Can the learning assistant dynamically adjust the content according to the learning progress? The core of these problems is not how personified the model is, but whether the entire memory system is sufficiently engineered, operational, and scalable.

However, to really solve these pain points, it is far from enough to rely on scattered memory functions. A complete solution designed for the production environment from access, use, operation and maintenance, and compliance is needed. AgentLoop MemoryStore starts from the real pain points of enterprises and uses a set of out-of-the-box, flexible, open, stable and reliable memory system to turn "usable" agents into "daring and easy-to-use" agents.

Out-of-the-Box: No Duplication of Infrastructure Construction, so That Memory Capabilities Directly Into the Existing Business
Many teams are not unable to make Memory Demo, but are stuck in the access cost. A self-built memory system often means that you must simultaneously process vector storage, structured storage, model invocation, asynchronous tasks, monitoring and alerting, permission isolation, and SDK encapsulation. Technically, it is not impossible, but the pace of product launch will be seriously slowed down. The first value of AgentLoop MemoryStore is not how cool the feature is, but how convenient it is:

a. out-of-the-box: you do not need to create a self-built vector database, MSMQ, or background task system. you can activate it and use it in a one-stop manner. it provides the ability to write and store raw data to long-term memory recall. Enterprise agents only need to focus on their own agent development, without the need to focus on the complex memory extraction process.

b. Multiple docking solutions: It provides a complete API and SDK for data writing and memory recall. The client can be seamlessly connected. In addition, AgentLoop MemoryStore allows you to consume trace data collected by observable probes. You only need to load the probes in the program to collect user interaction information in a non-intrusive manner without modifying the original business logic. For teams with existing memory-related code, the product is also compatible with the Mem0 API, enabling zero-cost migration. In addition, it also supports multiple access forms such as MCP Server and OpenClaw plug-ins, which can be easily integrated into various mainstream Agent frameworks, allowing existing systems to quickly have long-term memory capabilities.

c. Cross-device memory sharing: provides SaaS hosting services. Memory sharing is supported across machines, instances, and sessions. Compared with the open-source standalone memory system, AgentLoop Memory provides memory sharing across devices. In an enterprise-level agent, the agent generally runs in a sandbox for permission isolation. If the memory system is a stand-alone version, it will disappear with the destruction of the agent instance. However, based on AgentLoop Memory, the agent instance can be destroyed at any time, but the memory can be forever.

Business Scenario Example: Intelligent Customer Service
A typical customer service Agent, most afraid of is "talked yesterday, today all forget". The user explained the order problem, receiving preference and communication habits yesterday. When entering the line again today, the system started asking questions from scratch and the experience would collapse immediately. After you connect to the AgentLoop MemoryStore, the customer service team does not need to rewrite the entire memory logic. Mem0-compatible interfaces or OpenClaw plug-ins can be used to recall and write memories into existing processes. When users consult again, Agent can first see key information such as "last ticket progress", "users' common addresses" and "preferred communication methods". Naturally, answers are more continuous and manual transfer is more efficient. Compared with many open source memory solutions that are more suitable for local experiments or single-machine deployment, the SaaS-based AgentLoop MemoryStore also has a very practical advantage: memory is not tied to a single machine, but can be continuously shared among different devices, different instances, and different service nodes. If the user communicates with the Agent on the web page in the morning and moves to the mobile terminal in the afternoon, or the request is routed to another machine, the system can still continue the same memory. This cross-machine sharing capability is closer to the way enterprises operate real online services.

The focus of this type of value is not "technically achievable", but "how long the business team can use it". For many enterprises, going online as soon as a week is often more meaningful than one more concept function.

Flexible and Open: Memory Is Not Only Stored, but Also Supports Business Processing and Precise Retrieval
After solving the problem of "fast access", the next key is to make the memory really fit the business, rather than simply piling up historical conversations. Memory is prone to homogenization because many products only solve the "storage" problem, but do not really solve the "how to remember, what to remember, when to take" problem. In an enterprise scenario, memory is never a static file, but a set of dynamic assets that are updated with business changes. The core difference of AgentLoop MemoryStore is that it is open enough to "memory processing" and "memory retrieval": it supports multi-dimensional memory extraction, not only retains the original dialogue content, but also automatically extracts structured memories such as user preferences, factual information, and scene summaries, so that memories are no longer scattered chat records. At the same time, it supports the dynamic update of memory rather than a mere addition, when the user's preference changes, the system will automatically update the old memory, from the source to reduce the accumulation of dirty data. It also supports flexible custom rules, whether it is the global extraction policy of the entire memory base or the special processing rules of a single message, which can be flexibly defined according to business requirements, so that the memory fully fits your business logic. In addition, it also provides a hierarchical retrieval strategy from L1 to L3, covering basic hybrid retrieval, refined Rerank to deep Agetic Search, taking into account the response speed, recall accuracy and deep semantic understanding capabilities in all aspects. The most important point here is that enterprises do not have to accept a "black box Memory" default understanding, but can inject their own business judgment into it.

Business Scenario: Sales Assistant
The key memory in the sales scenario is often not a "customer is interested in the product", but more detailed structured information: the current procurement stage of the customer, who is the decision maker, whether the budget is approved, what objections were raised in the last phone call, and what actions were agreed next. If you just put all the chat records back into context, the cost is high, the noise is much, and the effect is not stable. A more effective way is to extract information such as "organizational structure", "business opportunity stage", "historical objection" and "next action" into renewable long-term memory, and then cooperate with hierarchical retrieval to recall only the most relevant parts in the current round. In this way, Agent gives not only a "chat" reply, but more like a sales colleague who has really followed up the customer process.

Business Scenario: Learning Assistant
In the learning scene, the more memory, the better. The system needs to distinguish between "long-term stable learning goals" and "short-term changes in knowledge mastery". For example, a user prefers video explanation at the beginning and then makes it clear that he prefers topic-driven learning. Another example is that after several rounds of practice, the old memory should be corrected instead of being kept as "weak points in learning".

AgentLoop MemoryStore supports separate processing by memory type and extraction strategy, allowing Learning Assistant to not only remember users, but also "remember changes." This improvement of the personalized experience is often more direct than simply expanding the context window.

Serverless, Elastic, and O&M-Free: Memory Does Not Act as a System Bottleneck and Does Not Add Infrastructure Burden
Memory function is easy to use, flexible is not enough, once on the production, stability and operation and maintenance costs become the key to determine whether the landing. Once Memory enters the production environment, the real test is often not "whether it can be extracted", but "whether the main link will be slowed down during high concurrency". Many solutions work well in the Demo phase, but problems will be exposed when they reach the real business traffic: synchronous extraction is too slow, call queuing, upstream and downstream timeout, resource expansion depends on manual work, and monitoring and alerting are not systematic. AgentLoop MemoryStore is designed to be "production-ready": It uses the memory pipeline architecture of asynchronous writing to process time-consuming memory retrieval in the background to minimize the impact on the main process. Relying on the data processing pipeline developed by AgentLoop, it can also perform multi-dimensional deduplication for large-scale interactive data, covering lexical deduplication, hash deduplication, and semantic vector deduplication, reducing redundant dirty data from the source. At the same time, it completely decouples the storage, calculation and retrieval modules. Each module can be expanded independently according to the actual load and can be easily adapted to the Auto Scaling capacity no matter how the business traffic fluctuates. In addition, it natively supports multi-tenant isolation, complete audit logs, and end-to-end observability to fully meet the O&M and compliance requirements of enterprises.

Business Scenario: Customer Service and Shopping Guide During the Promotion Period
When e-commerce is promoted, the pressure on customer service and shopping guide agents is usually several times or even dozens of times higher than usual. If the memory retrieval is executed in full synchronization, each dialogue has to wait for the model extraction and writing to be completed, and the latency of the main link will increase rapidly, eventually affecting the whole site experience. A more reasonable approach is to leave "the most critical recall to the user's reply" in the real-time path and put "more complex memory processing and precipitation" into the asynchronous pipeline. In this way, the Agent can respond in a timely manner without blocking the foreground service due to background memory processing. For enterprises, this is not a simple architecture optimization, but a question of whether they can stabilize service quality at critical moments.

The significance of Serverless and O&M-free is also here. What the enterprise team really wants to save is not only a few machines, but also a whole set of maintenance costs around Memory: expansion, monitoring, exception troubleshooting, task backlog, data isolation, and permission control. If you do all of this on your own, Memory will quickly go from being an "empowerment" to a "new burden."

Why AgentLoop Memory Is More Suitable for Production Environment: Not Only Can Remember, but Also Can Be Verified, Managed and Audited
The access is fast, flexible, and stable. Eventually, it must be quantifiable, controllable, and compliant before it can truly enter the core link of the enterprise. When enterprises choose Memory, they will not only look at the concept, but also look at the results. Don't look at the advertisement, look at the curative effect, whether the effect is good or not, go to Benchmark to run and see. Based on a unified Benchmark, it is the touchstone for measuring different Memory systems. In the Locomo Benchmark evaluation, the accuracy score of AgentLoop Memory reaches 84.07%. At the same time, compared with EverMemos, the recalled memory volume is 30% less. This means that it doesn't just "remember more", but gives more efficient hit results with less context overhead.

In addition to the effect, enterprises are also concerned about the long-term operation. AgentLoop MemoryStore also provides several capabilities that are critical to the production environment: in addition to the effect, enterprises are also concerned about long-term operation. AgentLoop MemoryStore also provides several critical capabilities for the production environment: it has built-in multi-tenant data isolation capabilities to meet enterprise-level security boundary requirements; it also provides complete audit logs to support the full tracking of memory additions, deletions, modifications, and checks to meet the requirements of compliance audits. It also supports comprehensive observability and cost analysis capabilities. You can easily view the latency, token consumption, request volume, and storage volume, and quickly troubleshoot problems. It also supports multiple integration methods and reduces the access threshold for different technology stacks.

In other words, it wants to deliver not just a "memory agent", but a memory infrastructure that enterprises can confidently incorporate into their core business links.

Best Practice: OpenClaw + AgentLoop MemoryStore - Low-threshold Access to Long-term Memory
To enable more teams to use reliable long-term memory, OpenClaw is further integrated with AgentLoop MemoryStore. This allows developers to quickly provide stable, reusable, and operational enterprise-level memory capabilities to existing agents without the need to build memory modules from scratch. If you are already using OpenClaw, the cost of accessing AgentLoop MemoryStore will be lower. We have packaged the integration solution as a separate npm package openclaw-plugin-agentloop-memory that, once installed and configured, can add enterprise-class long-term memory to OpenClaw without modifying the OpenClaw code itself.

Prerequisites
Before you perform the migration, make the following preparations:

■ You have an Alibaba Cloud account and have activated the AgentLoop MemoryStore service.

■ Create a Workspace and MemoryStore in the AgentLoop MemoryStore console

■ The AccessKey ID and AccessKey secret of your Alibaba Cloud account.

Installation
Execute in the OpenClaw project directory:

npm install openclaw-plugin-agentloop-memory
Configure
After the installation is complete, enable the plug-in in the OpenClaw configuration and specify the connection parameters. Typical configurations are as follows:

{
  "memory-agentloop": {
    "endpoint": "cms.cn-hangzhou.aliyuncs.com",
    "accessKeyId": "${ALIBABA_CLOUD_ACCESS_KEY_ID}",
    "accessKeySecret": "${ALIBABA_CLOUD_ACCESS_KEY_SECRET}",
    "workspace": "my-workspace",
    "memoryStore": "my-memory-store"
  }
}

The following table describes the core parameters :

■ endpoint: the API endpoint address of AgentLoop MemoryStore. Enter the endpoint address based on the region where the instance is located, for example, cms.cn-hangzhou.aliyuncs.com

■ accessKeyId /accessKeySecret: Alibaba Cloud access credential, supports environment variable injection to avoid plaintext storage

■ workspace: Name of the workspace created in the AgentLoop MemoryStore control

■ memoryStore: The name of the memory bank in the workspace.

The plug-in also provides the following optional configurations:

■ userId /agentId: used for user-level and agent-level data isolation, applicable to multi-tenant scenarios

■ autoCapture: On by default, it automatically extracts valuable information from the conversation and writes it to the memory bank.

■ autoRecall: On by default, it automatically retrieves relevant memories and injects context before each conversation starts.

■ inferOnAdd: This feature is enabled by default. Intelligent extraction is enabled when you write data to the memory. Multi-dimensional memory extraction and deduplication are automatically performed.

Capabilities provided by the plug-in
After installation, the plug-in adds three types of capabilities to OpenClaw:

■ Agent tools: three memory operation tools: registration memory_recall, memory_store and memory_forget, which are convenient for Agent to actively retrieve, write and delete memory during dialogue.

■ Automated hooks: When autoRecall and autoCapture are enabled, memory recall and asynchronous precipitation are automatically completed to reduce business code transformation.

■ CLI command: provides openclaw agentloop command line capabilities to facilitate developers to search, add, list, and delete memories directly in the terminal, and perform connectivity checks.

SDK for Python Quick Experience Demo
If you want to quickly verify the effect first, you can also experience it directly through the Python SDK:

1.Get AgentLoop Memory SDK

pip install agentloop-memory
2.Run the sample program

from agentloop_memory import Config
from agentloop_memory.client import AgentLoopMemoryClient
import os
import time
def main():
    # 1. Init memory store client
    config = Config(
        access_key_id=os.getenv("ALIYUN_ACCESS_KEY_ID"),
        access_key_secret=os.getenv("ALIYUN_ACCESS_KEY_SECRET"),
        endpoint=os.getenv("CMS_ENDPOINT", "cms.cn-shanghai.aliyuncs.com"),
    )
    client = AgentLoopMemoryClient(
        config,
        workspace=os.getenv("CMS_WORKSPACE"),
        memory_store=os.getenv("CMS_MEMORY_STORE"),
    )
    # 2. Create memory store
    result = client.create_memory_store(
        description="Example memory store",
        extraction_strategies=["FACT"],
    )
    print("create_memory_store:", result)
    time.sleep(5)
    # 3. Add memory
    result = client.add(
        messages="I live in Hangzhou and love visiting West Lake",
        user_id="user123",
    )
    print("add:", result)
    time.sleep(120)
    # 4. Search memory
    result = client.search(
        query="Where do I live?",
        user_id="user123",
    )
    print("search:", result)
    # 5. Get all memories
    result = client.get_all(
        user_id="user123",
        page=1,
        page_size=10,
    )
    print("get_all:", result)
    # 6. List memory stores
    result = client.list_memory_stores(max_results=10)
    print("list_memory_stores:", result)
if __name__ == "__main__":
    main()

Sample result

{'status_code': 200, 'headers': {'server': 'AliyunSLS', 'content-length': '0', 'connection': 'keep-alive', 'access-control-allow-origin': '*', 'date': 'Mon, 02 Feb 2026 03:27:53 GMT', 'x-log-time': '1770002873', 'x-log-requestid': '698019B5FA0F42BA63073DF6'}}
{'results': [{'event_id': '800c03bc-dc54-42de-bd07-153421f88259', 'message': 'Memory processing has been queued for background execution', 'status': 'PENDING'}]}
{'results': [{'created_at': 1770002874, 'hash': '55566d2fdec59e0a3bf8870b1cb17bfd', 'id': '019c1c65-9745-7773-92f8-189a2b4a3721', 'memory': 'lives in Hangzhou, 'score': 0.5316177221048695, 'updated_at':: updated_at': 1770002874, 'user_id': 'user_0.46264787090919 ', '74 createdy': at': 177a' 1770002874, 'user_id': 'user123'}, {'created_at': 1770002874, 'hash': '7b869aba23294ab37679c5f7e7465921', 'id': '019c1c65-990e-7381-8ba4-794867a634bd', 'memory': 'like the scenery of hangzhou', 'score': 0.4317308740071, 'updated_at': 1770002874, ''user_id': 'user12l':} 3'
{'results': [{'created_at': 1770002874, 'hash': '55566d2fdec59e0a3bf8870b1cb17bfd', 'id': '019c1c65-9745-7773-92f8-189a2b4a3721', 'memory': 'Lived in Hangzhou, 'updated_at': 1770002874, 'user_id': upered': 'user12y', {'7b869aba23294ab37679c5f7e7465921' 'user123'}, 'hash': 170002874', 'hidat ', 'hash' 'hash' 'hash' 1770002874, 'hash': '939ed9d15f907d252363fd0e2cffb9a9', 'id': '019c1c65-9ac3-7cd1-afea-1f091dcdc6fe', 'memory': 'frequent visit to the West Lake ', 'updated_at': 1770002874, 'user_id': 'user123'}], 'relations': []}

After the memory is added, the system automatically extracts and stores three key pieces of information:

■ "I live in Hangzhou"

■ "Love the scenery of Hangzhou"

■ "I often go to the West Lake to play."

When querying "Where do I live?", the system will accurately return "live in Hangzhou" and return other associated memories based on the relevance. The whole process without manual annotation, memory extraction and retrieval can be done automatically.

Summary
Today's Memory market does not lack new concepts, but solutions that can really help enterprises run agents, run stably, and run out of business value. The focus of AgentLoop MemoryStore is not to make "memory" more mysterious, but to do the three most realistic things well: to connect to the existing system faster, to fit the specific business more flexibly, and to run in the production environment more carefully. For teams that are already doing customer service, sales, learning, shopping guide and other agents, such Memory is really worth seeing and being connected to the main link.

Don't let your agents have only seven seconds of memory. Immediate access to AgentLoop MemoryStore so that data is truly deposited into reusable business wisdom:

https://cmsnext.console.alibabacloud.com/agentloop/home

LoongCollector + ACS Agent Sandbox: Build a Production-grade AI Agent Runtime Platform

ObservabilityGuy — Tue, 26 May 2026 02:56:58 +0000

This article introduces AgentLoop MemoryStore, a fully managed, enterprise-grade memory solution designed to give AI Agents long-term, reliable memory for production environments.

1.Security and Observability Challenges of AI Agents
With the rapid development of Large Language Models (LLMs), AI Agents are moving from the lab to production. From intelligent customer service to code assistants, and from data analytics to automated O&M, AI Agents are transforming how we work. However, unlike traditional applications, AI Agents possess two distinct characteristics:

● Unpredictable behavior: The same input might generate different outputs and invoke different toolchains.

● Execution capability: Agents don't just "talk"; they "act"—accessing data, invoking APIs, and executing operations.

These two characteristics present entirely new challenges.

Core challenge 1: Runtime security (What are Agents permitted to do? Who defines the boundaries?)
Consider this scenario: A customer service Agent answering a query is subjected to a prompt injection attack. It accidentally accesses another user's order information, or even triggers a refund API. This is a real-world security risk, not science fiction.

AI Agent security risks primarily stem from two areas:

1.Lack of strong isolation in execution environments

Agents require data access and tool invocation at runtime. Without strict permission controls, prompt injections or accidental triggers can lead to unauthorized access, data leaks, or unintended operations—such as an Agent bypassing security checks to access a restricted database.

2.Lack of control over external capabilities

The greatest threats often arise from the abuse of external capabilities—such as abnormal outbound calls, SSRF/intranet probing, or sensitive data persistence and exfiltration. For example, an Agent might be tasked with "checking the weather" but actually initiates a scan of internal network services.

Core Challenge 2: Full-link Observability (What did the Agent do? Why did it do it? How effective was it?)
Traditional applications are deterministic; the same input yields the same output. AI Agents, however, may make different decisions each time, leading to three major observability hurdles:

1.Behavior is hard to reproduce and troubleshoot

For the same query, an Agent might use Tool A today, Tool B tomorrow, or simply provide a direct answer the day after. When errors occur, identifying the exact point of failure is difficult.

2.Difficulty in cost control and attribution

Costs are driven by LLM token consumption and external API calls, both of which fluctuate significantly. It is often unclear which users, tasks, or models are driving up expenses.

3.Quality is hard to measure and optimize

Output quality depends on model capability, prompt design, and retrieval data. Because these factors change constantly, it is difficult to pinpoint what is working, what isn't, and how to optimize.

Why Is a Specialized Solution Necessary?
Traditional monitoring and security solutions fall short in AI Agent scenarios:

This is why a runtime platform and observability solution specifically designed for AI Agents are essential. Let's explore how ACS Agent Sandbox and LoongCollector address these challenges.

2.ACS Agent Sandbox and LoongCollector: Comprehensive Security and Observability
ACS Agent Sandbox provides a secure execution environment based on Kubernetes, while LoongCollector acts as a telemetry data collector to provide agents with comprehensive monitoring and analysis. Together, their deep integration forms a complete production-grade execution platform for AI Agents.

2.1 ACS Agent Sandbox: Providing Runtime Security
Alibaba Cloud Container Service (ACS) Agent Sandbox is a specialized environment launched by Alibaba Cloud. Built on Kubernetes, it provides a secure, isolated, and scalable platform for running AI Agents.

2.2 LoongCollector: Providing Sandbox Observability
LoongCollector is a unified telemetry collector open-sourced by the Alibaba Cloud Observability team. Designed for cloud-native and high-performance scenarios, it offers unique advantages for AI Agent use cases:

Extreme Performance and Ultra-low Overhead
AI Agents are compute-intensive, so observability components must be lightweight to avoid impacting business operations:

● Zero-copy architecture: Utilizes Memory Arena and zero-copy to minimize unnecessary memory overhead.

● Event pooling and reuse: High-frequency object pooling reduces memory allocation and Garbage Collection (GC) pressure.

● High single-core throughput: A single core can support log collection throughput of up to 500 MB/s.

Unified Collection: Full Coverage of Logs, Metrics, and Traces
● Logs: Supports stdout/stderr and file logs; automatically associates Kubernetes metadata such as Pods, Namespaces, and Labels.

● Metrics: Native support for Prometheus Exporter, system metrics (CPU, memory, network, and disk I/O), and GPU metrics (NVIDIA DCGM).

● Traces: Full support for OpenTelemetry.

Edge Computing: Moving Processing to the Data Source
Beyond collection, it performs edge-side preprocessing to reduce transmission and storage costs:

● High-performance C++ plugins and Structured Process Language (SPL) engine.

● Supports complex processing: Filtering, transformation, and aggregation.

● Edge-side dimensionality reduction: Minimizing noise and data volume at the source.

Enterprise-Grade Reliability: Ensuring Zero Data Loss and Stable Operations
Data reliability

● At-least-once delivery semantics.

● Local disk caching: Persisting data to disk during network anomalies and retransmitting upon recovery.

● Automatic retry and exponential backoff.

● Backpressure and rate limiting: Protects the system during downstream congestion.

Operational reliability:

● Multi-tenant pipeline isolation.

● Priority scheduling: Ensuring critical data is processed first.

● Hot updates and graceful changes: Configuration changes take effect without restarts or service interruptions.

Unified Management for Large-Scale Elastic Scenarios
● ConfigServer: Centralized configuration management supporting tens of thousands of Agents.

● Remote configuration delivery: Changes take effect in real-time without requiring manual login.

● Status and performance monitoring: A unified view of health and resource overhead.

2.3 Deep Integration: LoongCollector Provides Zero-Intrusion, Automated, and Highly Reliable Observability for Sandbox

● ACS management automatically injects the LoongCollector container into the Sandbox.

● Via shared file path mounting.

● Use the Pod network to perform Prometheus scraping on AI Agents or receive OpenTelemetry data.

Through the deep integration of ACS Agent Sandbox and LoongCollector, we have built a comprehensive production-grade platform for AI Agents:

3.Running OpenClaw Using ACS Agent Sandbox and LoongCollector
OpenClaw is a trending AI application that redefines the boundaries of AI assistants. Its core value is no longer just answering questions, but understanding intent, planning steps, and invoking tools to complete tasks—acting as an "always-on" digital employee. Next, let's explore how to run OpenClaw securely and with full observability using ACS Agent Sandbox and LoongCollector.

*3.1 Enabling Sandbox LoongCollector Injection for ACK and ACS Clusters
ACK clusters
*
Note: Install the following components in advance:

● Install the LoongCollector component in Components and Add-ons.

● Install the ACK Virtual Node component in Components and Add-ons.

● Install ack-agent-sandbox-controller components in Components and Add-ons.

● To expose services via EIP, install the ack-extend-network-controller component from the Marketplace. Refer to the help document for specific configuration steps.

Modify the eci-profile ConfigMap in the kube-system namespace. The slsMachineGroup parameter defines the Sandbox machine group identifier; we recommend using a unique identifier different from the ACK DaemonSet group.

ACS clusters

Note: Install the following components first:

● Go to Components and Add-ons and install the ack-agent-sandbox-controller component (version ≥0.5.3).

● To expose services via EIP, go to Components and Add-ons in the ACK cluster and install the ack-extend-network-controller component.

● Go to Components and Add-onsand install the in alibaba-log-controller component.

The machine group identifier is the unified ACS cluster group ID: k8s-log-${cluster_id}

3.2 Deploying OpenClaw in ACS Agent Sandbox
Enable the OpenTelemetry (OTel) plugin for OpenClaw

Note

● Ensure extensions/diagnostics-otel is included when packaging the OpenClaw image.

● You must enable diagnostics-otel in the configuration to report metrics and trace data.

Configure ~/.openclaw/openclaw.json

Note: The endpoint configured here will be required for the LoongCollector collection configuration later.

{  
  "plugins": {  
    "allow": ["diagnostics-otel"],  
    "entries": {  
      "diagnostics-otel": { "enabled": true }  
    }  
  },  
  "diagnostics": {  
    "enabled": true,  
    "otel": {  
      "enabled": true,  
      "endpoint": "http://127.0.0.1:4318",  
      "protocol": "http/protobuf",  
      "serviceName": "openclaw-gateway",  
      "traces": true,  
      "metrics": true,  
      "logs": true,  
      "sampleRate": 1,  
      "flushIntervalMs": 60000  
    }  
  }  
}

OpenClaw sandbox deployment example

Below is a simplified example of creating an OpenClaw sandbox directly using a Sandbox CR:

apiVersion: agents.kruise.io/v1alpha1  
kind: Sandbox  
metadata:  
  name: openclaw  
  namespace: default  
spec:  
  template:  
    metadata:  
      labels:  
        alibabacloud.com/acs: 'true'  
        app: openclaw  
    spec:  
      containers:  
        - name: openclaw  
          # Replace with the actual OpenClaw image address  
          image: <open-claw image address>   
          imagePullPolicy: IfNotPresent   
          resources:  
            limits:  
              cpu: '4'  
              memory: 8Gi  
            requests:  
              cpu: '4'  
              memory: 8Gi  
          securityContext:  
            readOnlyRootFilesystem: false  
          terminationMessagePath: /dev/termination-log  
          terminationMessagePolicy: File  
      dnsPolicy: ClusterFirst  
      paused: true  
      restartPolicy: Always  
      schedulerName: default-scheduler  
      securityContext: {}  
      terminationGracePeriodSeconds: 1

3.3 Full Observability Collection Configuration
As described in Is Your OpenClaw Really Running Under Control?, the observability data for OpenClaw is as follows:

Session logs

apiVersion: telemetry.alibabacloud.com/v1alpha1  
kind: ClusterAliyunPipelineConfig  
metadata:  
  name: openclaw-session-log  
spec:  
  config:  
    aggregators: []  
    global: {}  
    inputs:  
      - Type: input_file  
        # This path varies depending on the run path of the openclaw image.  
        FilePaths:  
          - /home/node/.openclaw/agents/main/sessions/*.jsonl  
        MaxDirSearchDepth: 0  
        FileEncoding: utf8  
        EnableContainerDiscovery: true  
        # Filter containers based on the OpenClaw sandbox information.  
        ContainerFilters:  
          K8sPodRegex: ^(openclaw.*)$  
    processors:  
      - Type: processor_parse_json_native  
        SourceKey: content  
    flushers:  
      - Type: flusher_sls  
        Logstore: openclaw-session-log  
    sample: ''  
  # Replace this with the sandbox machine group name of the ACK or ACS cluster.  
  machineGroups:  
    - name: <your-sandbox-machine-group>  
  # The project to which logs are collected.  
  project:  
    name: k8s-log-xxx  
  # The Logstore to which logs are collected.  
  logstores:  
    - name: openclaw-session-log

Application logs

apiVersion: telemetry.alibabacloud.com/v1alpha1  
kind: ClusterAliyunPipelineConfig  
metadata:  
  name: openclaw-app-log  
spec:  
  config:  
    aggregators: []  
    global: {}  
    inputs:  
      - Type: input_file  
        FilePaths:  
          - /tmp/openclaw/*.log  
        MaxDirSearchDepth: 0  
        FileEncoding: utf8  
        EnableContainerDiscovery: true  
        # Filter containers based on OpenClaw sandbox information.  
        ContainerFilters:  
          K8sPodRegex: ^(openclaw.*)$  
    processors:  
      - Type: processor_parse_json_native  
        SourceKey: content  
    flushers:  
      - Type: flusher_sls  
        Logstore: openclaw-app-log  
    sample: ''  
  # Replace this with the name of the sandbox machine group for your ACK or ACS cluster.  
  machineGroups:  
    - name: <your-sandbox-machine-group>  
  # The destination project for data collection.  
  project:  
    name: k8s-log-xxx  
  # The destination Logstore for data collection.  
  logstores:  
    - name: openclaw-app-log

OpenTelemetry

apiVersion: telemetry.alibabacloud.com/v1alpha1  
kind: ClusterAliyunPipelineConfig  
metadata:  
  name: openclaw-otel-config  
spec:  
  config:  
    # This corresponds to the logstores below. It distributes and stores OpenTelemetry logs, metrics, and trace data.  
    aggregators:  
      - Type: aggregator_opentelemetry  
        MetricsLogstore: openclaw-otel-metrics  
        TraceLogstore: openclaw-otel-traces  
        LogLogstore: openclaw-otel-logs  
    global: {}  
    inputs:  
      - Type: service_otlp  
        Protocals:  
          HTTP:  
            # Corresponds to the diagnostics-otel Endpoint enabled in OpenClaw.  
            Endpoint: '127.0.0.1:4318'  
            ReadTimeoutSec: 10  
            ShutdownTimeoutSec: 5  
            MaxRecvMsgSizeMiB: 64  
    processors: []  
    flushers:  
      - Type: flusher_sls  
        Logstore: openclaw-otel-logs  
  # Replace with the Sandbox machine group Name for the ACK or ACS cluster.  
  machineGroups:  
    - name: <your-sandbox-machine-group>  
  # The project for Collection.  
  project:  
    name: k8s-log-xxx  
  # The Logstore for Collection. Note that OpenTelemetry has three Data Types. You must define three Logstores.  
  # For metrics Data, set telemetryType to Metrics.  
  logstores:  
    - name: openclaw-otel-logs  
    - name: openclaw-otel-metrics  
      telemetryType: Metrics  
    - name: openclaw-otel-traces

3.4 Summary: Fully Resolving OpenClaw Security Challenges
Sandbox runs OpenClaw securely and in isolation

● Each Sandbox runs in an isolated kernel environment, preventing malicious code from attacking host system programs.

● Each Sandbox uses an isolated temporary file system to prevent unauthorized reading, tampering, or deletion of host files.

LoongCollector enables full-stack observability for OpenClaw

4. Summary and Outlook
The production-readiness of AI Agents is not a matter of "if," but "how." Security and observability are not optional—they are essential requirements.

If you are building an AI agent application:

● Start now by prioritizing runtime security and observability.

● Choose the right tools instead of reinventing the wheel.

● Establish best practices and promote them within your team.

● Continually learn and optimize to ensure your Agents create real value.

Both ACS Agent Sandbox and LoongCollector are open platforms; we invite you to try them and share your feedback. Together, let's build a more secure, reliable, and efficient production environment for AI Agents. We hope this article provides valuable reference and inspiration for your observability journey.

Human-Robot Half Marathon: The Large-Scale O&M Challenge for Embodied Intelligence Beyond the Racecourse

ObservabilityGuy — Wed, 20 May 2026 02:38:23 +0000

This article introduces an Alibaba Cloud-powered O&M observability system tackling humanoid robot challenges in large-scale, outdoor, and long-distance scenarios.

A special half marathon has just concluded in Beijing. More than 300 humanoid robots competed alongside humans, vying across dimensions such as autonomous navigation, dynamic balance, and multi-robot coordination, setting a global record for the scale of human-robot co-running events. When hundreds of robots collectively run 21 kilometers, what we see is not just a race, but a large-scale public stress test for the realm of embodied intelligence. As the race ends, a bigger challenge has emerged beyond the racecourse—

In the face of new embodied intelligence scenarios characterized by clustering, mobility, and complexity, the industry urgently needs a standardized, reusable, integrated O&M system that adapts to outdoor weak-network and multi-device heterogeneous environments. Leveraging Alibaba Cloud's full-spectrum observability capabilities, with Simple Log Service (SLS), CloudMonitor (CMS), and Application Real-Time Monitoring Service (ARMS) as the core foundation, a collaborative O&M observability system for humanoid robots has been built. This system precisely matches the requirements of typical scenarios involving long-distance movement, multi-robot formation coordination, and full environment variable interference, providing a practical reference for the industry to solve large-scale O&M challenges.

Three Dilemmas: New Challenges in Embodied Intelligence O&M Observability
The 21-kilometer open course of the half marathon is an extreme stress test of the comprehensive stability of humanoid robots. It also exposes the three core bottlenecks in deploying embodied intelligence clusters at scale — a common challenge across all outdoor large-scale scenarios.

● Environmental uncertainty is the primary challenge of outdoor operations. In open scenarios, temperature, humidity, and lighting conditions change in real time, while uncontrollable factors such as road bumps, ramps, curves, pedestrian crossings, and wireless signal fluctuations persist, continuously interfering with sensor detection accuracy, communication transmission stability, and power system payload balance. Especially under high-temperature conditions, prolonged high-load operation of robot active joints, computing power modules, and battery components accelerates hardware aging and significantly increases component failure rates. Device operation remains in a state of Dynamic fluctuation, where a single environmental disturbance can trigger cascading abnormalities.

● Hidden damage and coupling threats from highly integrated devices further amplify operational risks. Humanoid robots tightly integrate motion modules, multiple sensor types, edge computing, AI inference, wireless communication, and other multilayer systems with precise structure and high interdependency. Minor vibrations and low-speed collisions during movement do not cause obvious skin damage but can easily lead to irreversible hidden issues such as slight displacement of lidar and vision cameras, loose joint wiring, and micro-deformation of internal support structures, which in turn cause navigation and obstacle avoidance inaccuracy, intermittent signal breaks, task execution bias, and other problems. Combined with individual device differences introduced by manual assembly, a minor abnormality in one device can quickly propagate to the entire formation, causing coordination disorder, rhythm desynchronization, and even cluster-level security risks.

● Traditional O&M patterns are completely unable to adapt to new scenarios. Previously, fixed devices relied on post-incident emergency repair, manual offline troubleshooting, and standalone independent management — a passive pattern with delayed response, entirely unsuitable for humanoid robots that operate with Dynamic mobility, all-weather jobs, and multi-robot collaboration. To support stable operation of large-scale clusters, it is essential to break down data silos among hardware indicators, system logs, algorithm links, and environmental data, move beyond experience-based manual O&M, and complete the transformation from passive remediation to active defense through full-dimension status visualization, proactive threat prediction, and rapid abnormal loss containment.

Cloud-edge Collaborative Data Collection Adapted to the Core O&M Features of Humanoid Robots
Based on the natural properties of humanoid robots — large-scale movement, unstable network environments, multi-brand heterogeneity, and long-duration continuous operations — the ideal O&M architecture for the industry must balance low-latency edge self-healing with cloud-based global unified management. By adopting a Layer 3 cloud-edge collaborative design spanning terminal body, edge gateway, and cloud platform, the solution reasonably separates the responsibilities of data collection, local management, computing power processing, and global analysis. Built around the three core O&M modules of real-time status monitoring, intelligent failure prediction, and hierarchical emergency response, Alibaba Cloud observability products form a complete capability matrix integrating indicators, traces, and logs to address industry pain points such as fragmented embodied device logs, difficulty in quantifying hardware indicators, and difficulty in troubleshooting hidden algorithm faults.
At the data access layer, the solution provides two highly available and flexible deployment modes to adapt to different outdoor conditions and network environments.

● The lightweight LoongCollector and Simple Log Service software development kit direct collection mode features extremely low resource usage on the device side and high compression and transmission efficiency. It meets high real-time monitoring requirements and supports dynamic adjustment of collection policies from the cloud, eliminating the need for frequent OTA upgrades on devices. LoongCollector is a new-generation Database Collector launched by Alibaba Cloud Simple Log Service that integrates performance, stability, and programmability. It extends and integrates the observability technology stack, breaking the single-scenario limitations of traditional log collectors, and supports the collection, processing, ingress, and sending of Logs, Metrics, Traces, Events, and Profiles.

● Based on the S3 protocol + Simple Log Service architecture, this mode is suitable for weak network and intermittent connectivity scenarios. Data is cached and encrypted locally and uploaded during off-peak hours. It is low-cost, highly reliable, not attached to a single vendor, and more extensible.

Both modes are fully compatible with 5G, Wi-Fi, IoT, and other communication methods, fully adapting to the complex and dynamic network environment of mobile robots.

Full-Domain, All-Dimension Observability for a Transparent Robot Cluster Operation System

Whether for outdoor formation movement or routine commercial deployment, the foundation for stable operation of large-scale embodied intelligence clusters lies in full-dimension, full-epoch, and full-link observability.

● At the hardware level, core indicators such as joint motor payload, current temperature, power supply health status, compute unit resource usage, inertial navigation calibration accuracy, sensing device data streams, sensor readings, and network quality are continuously collected to fully grasp the health status of core components and detect hardware threats such as overload, overheating, abnormal power supply, and sensor attenuation in advance.

● At the business and algorithm level, the running status of underlying core processes is monitored in real time, and various management events are managed at different levels, with a focus on intercepting faults and fatal exceptions. Key indicators such as perception and decision inference latency, path planning efficiency, and collaborative execution success rate are continuously tracked to fully restore algorithm running health and detect performance degradation and logical exceptions in a timely manner.

● At the scenario and environment level, full-epoch job info, device running status transitions, outdoor temperature and humidity environment data, physical collision management events, and other real-scene information are recorded. Through multi-dimension data cross-referencing, different failure root causes such as environmental interference, mechanical damage, algorithm bugs, and human operations are quickly distinguished, providing an objective basis for daily O&M and post-event review.

For the above observation scenarios, the three core dimensions of indicator monitoring, Tracing Analysis, and log administration are built in depth to form a full-coverage, strongly collaborative, and closed-loop global observability capability, targeting industry pain points such as invisible operation of embodied devices, difficulty in detecting exceptions, and difficulty in tracing failures.

● Indicator monitoring focuses on the model training realm, covering full-dimension timing monitoring and visualization management of AIBoost cluster AI infrastructure. Through continuous statistics on training resource payload, hardware conditions, environment parameters, and cluster running status, the training procedure can be quantified and abnormal threats can be warned in advance, ensuring the stability and reliability of AI model iteration from the ground up.

● Tracing Analysis provides deep, end-to-end visibility into service operations, enabling full-link visualization and tracing across the CDN mapping system, motion control services, AI inference links, and cross-device interface interactions. It accurately captures hidden application layer failures such as algorithm drift, background service stuttering, remote instruction blocking, and multi-machine collaborative scheduling conflicts, making previously invisible software and algorithm issues fully transparent and significantly improving the efficiency of troubleshooting soft abnormal issues.

● Log Administration: provides unified collection and standardized administration of end-to-end logs, including hardware operational logs, system process logs, AI module operation records, edge node management events, and job operation traces. It effectively addresses the challenges of scattered logs from heterogeneous devices, inconsistent formats, fragmented data, and difficulty in correlating and tracing issues. With high-throughput ingestion and second-level retrieval capabilities, it delivers complete, objective, and verifiable data support for failure review, root cause analysis, accountability determination, and batch issue tracing.

With global visualization and management capabilities, you can gain a macro-level view of overall cluster status, device online status, and overall payload fluctuations, while also drilling down into individual device details, achieving bidirectional integration between macro management and micro-level positioning. Combined with dynamic thresholds and intelligent anomaly detection, real-time alerts are triggered for high-frequency threats such as sudden power drops, high-temperature overloads, network disconnections, and data drift, enabling true proactive threat prevention and control.

Multi-Field Dependency Analysis to Resolve Incremental Hidden Threats with Predictive O&M
Compared with obvious hardware corruption, the slow attenuation of sensor accuracy, line contact fatigue, chronic component aging, algorithm performance degradation, and hidden structural hazards caused by long-term vibration are the key factors affecting the long-term stable operation of humanoid robots. Such progressive issues cannot be detected through manual inspection and require multi-source data field dependency analysis to implement data-driven predictive O&M.

Leveraging full-volume timing indicator data, this capability accumulates long-term insights into basic resource O&M, model training and inference efficiency evaluation, device payload changes, environmental impact patterns, and hardware aging trends to form a quantifiable health assessment baseline. Through end-to-end Tracing Analysis, the complete flow logic of instruction routing, service invocation, and algorithm computation is fully restored to quickly locate coordination bottlenecks and program anomalies. Combined with unified log administration, system events, error records, environmental changes, and external interference before and after an anomaly are correlated to fully reconstruct the failure scene.

Multi-dimension data association and cross-validation enable accurate discovery of potential patterns in device operation and early detection of hidden risks. Combined with a tiered alerting mechanism that filters invalid fluctuations and duplicate alerts, threats are escalated and handled by tiering. During the early stage of failure emergence, proactive intervention through parameter automatic rotation tuning, run policy optimization, and remote fine-grained control effectively extends the stable operation epoch of devices, reducing failure rates and burst maintenance costs at the source.

The deeper value of observability goes beyond ensuring current stable operation — it uses data from real, complex scenarios to feed back into product R&D and process upgrades, paving the way for long-term commercialization of humanoid robots. By leveraging comprehensive data accumulation, you can horizontally compare operational differences across devices of the same model and batch, quickly identify common issues caused by component batch bugs, schema design shortcomings, and manual assembly process bias, and help manufacturers optimize supply chains and production flows. Through quantitative analysis of algorithm performance, component payload, and sensing stability under different operating conditions, hardware limitations and algorithm bottlenecks are precisely distinguished, helping R&D teams optimize motion control, autonomous navigation, and coordination policies in a targeted manner.

Meanwhile, massive scenario data such as real road conditions, crowd interference, complex lighting, extreme temperature and humidity, and collision anomalies can continuously enrich the simulation training sample library, narrow the gap between the simulation environment and real outdoor scenarios, accelerate algorithm iteration and real-machine adaptation efficiency, and enable humanoid robots to move faster from competition demonstration scenarios to normalized, large-scale deployment.

Tiered Closed-Loop Emergency Response System for High Fault Tolerance Operation Assurance in Complex Scenarios
Open outdoor scenarios inherently involve uncertainty. Instantaneous environmental changes, accidental mechanical disturbances, and short-term network anomalies cannot be completely eliminated. A standardized, tiered, and automated emergency response mechanism is the key line of defense for ensuring continuous and stable cluster operation. Based on the business characteristics of multi-robot formation operation, a comprehensive three-level failure handling logic is established: minor individual anomalies, local coordination failures, and systemic major failures. O&M resources are reasonably allocated through tiered control to avoid excessive response or delayed handling.

When an abnormal event occurs, leverage the observability system to quickly locate the root cause: troubleshoot algorithm and schedule issues through business trace analysis, pinpoint the scope of hardware, power supply, and network anomalies using timing indicators, and restore the complete on-site context with full logs, significantly reducing failure troubleshooting and fix time. After each abnormal event is handled, the complete failure timeline, alerting records, root cause conclusions, and handling reports are automatically accumulated and archived. This not only forms an O&M closed loop, but also builds reusable practical experience for optimizing handling policies and iterating management rules for similar scenarios in the future.

Summary and Outlook
The Beijing Yizhuang Humanoid Robot Half Marathon vividly demonstrates the rapid rise of China's humanoid robot industry and clearly signals that clustering, outdoor operation, and scenario-based deployment are the inevitable direction for the future development of embodied intelligence. As hardware integration and AI algorithms continue to break through, O&M capabilities are becoming a key variable that widens the industry gap. Multi-robot collaboration, hidden threat prevention, and full lifecycle management in open and complex environments are common challenges that all humanoid robot companies must address.

Alibaba Cloud's full-domain observability solution for embodied intelligence, built on a cloud-edge collaboration architecture, integrates three core capabilities: indicator monitoring, Tracing Analysis, and log analysis. It fully addresses the scenario features of humanoid robots, including mobile operations, cluster formation, weak network adaptation, and long-duration runs. Rather than being limited to a single event application, it provides a mature, standardized, and replicable O&M capability frame for similar outdoor cluster, dynamic operation, and large-scale deployment scenarios across the industry.

In the future, as the mass production scale of humanoid robots continues to expand and application scenarios keep extending, data-driven artificial intelligence for IT operations, proactive predictive protection, and full-link observability systems will become the core foundation for high-quality development of the embodied intelligence industry, continuously helping China's humanoid robot technology advance from technical demonstration to full-scale commercial deployment.

Related Products
Simple Log Service: https://www.alibabacloud.com/en/product/log-service
CloudMonitor: https://www.alibabacloud.com/en/product/cloud-monitor

Put a Microscope on Hermes: Full Visibility into Agent Execution

ObservabilityGuy — Wed, 20 May 2026 02:26:18 +0000

Alibaba Cloud's OpenTelemetry-based observability plugin brings full visibility to Hermes AI agent execution, enabling traceable costs, performance, and security auditing.

Hermes is an autonomous AI agent runtime frame developed by Nous Research. Rather than a one-shot Q&A pair-style model encapsulation, it is an agent runtime that continuously runs, invokes tools, accumulates experience, and grows throughout the usage procedure.

When an AI agent truly starts solving a problem — whether it completes correctly or exhibits bias — the real challenge is often not whether the result is right, but what exactly it did.

A single run of Hermes is not an ordinary model invocation. A seemingly simple interaction may involve multiple rounds of inference, tool calling, result reinjection, context expansion, and new inference loops. The model decides whether a tool is needed for the next step, and tool results in turn affect the subsequent inference path. Cost, latency, and faults often occur in the middle of this procedure.

If the system can only provide a final reply, a few scattered logs, or a usage summary for a single invocation, Hermes remains a black box. You know it completed the job, but you can hardly tell how. You know the request consumed a lot of tokens, but you can hardly tell which step drove up the cost. You know the user experience has slowed down, but you can hardly determine whether model generation slowed, tool execution went abnormal, or ReAct (Reasoning + Acting) loops spiraled out of control.

This is exactly our starting point for building observability into Hermes.

This article introduces a set of observability plugin solutions provided by Alibaba Cloud for Hermes. It can revert the real execution procedure of Hermes into a structured invocation chain: where a session starts, how many rounds of inference it goes through, which tools are invoked, how many tokens are spent, which step is the most time-consuming, and at which edge zone a fault occurs. Which operations are malicious, and how much sensitive data has been leaked.

If you are using Hermes for real-world jobs, you will almost certainly encounter these problems:

● Why is it so expensive this time?

● Why is it so slow this time?

● Did it actually invoke that tool?

● Did the tool it used leak data?

What these problems have in common is that they are not "results" but "procedures". So, if we can only see the last reply, then from an observational point of view, Hermes is still not interpretable.

What Exactly Are We Trying to Solve
The Alibaba Cloud Hermes observability plugin focuses on solving the following four types of problems.

The first is that the procedure is invisible.

After integrating an LLM, many systems still only show user input, final output, and a usage summary. But the real run of Hermes is far more than that. Behind a single response, there may be multiple rounds of inference, multiple tool executions, continuous context expansion, and new inference loops. Without a call chain, the intermediate procedure is essentially empty. The first thing we did was fill in that gap.

The second is that costs are not attributable.

The token bill itself isn't the hardest problem — the hardest part is not knowing where the money actually goes. A Hermes run can be expensive because the context suddenly explodes in a certain round, a tool returns an oversized result, the final round produces overly long output, or a certain class of jobs naturally triggers more steps. Without visibility into the tokens for each round of model invocation, cost analysis is nothing more than guesswork.

The third category is that performance cannot be broken down.

Users will only tell you "it's getting slower," but "slow" by itself carries no useful info. What you really need to distinguish is: is the first token slow, or is overall generation slow? Is tool execution slow, or is multi-round ReAct inference itself running too long? Only by separating these stages can a "slowdown" become a problem you can actually pinpoint.

The fourth category is that results cannot be reviewed.

Often the hardest issues to deal with are not clear-cut faults, but cases where "it looks like it succeeded, but the result is wrong." This is very common in agent systems: Hermes invokes the wrong tool, the tool returns incomplete results, Hermes continues to infer based on partial info, and ultimately produces an answer that seems reasonable on the surface but has already gone off track. Without traces, post-mortem review is nearly impossible. With traces, the problem shifts from "guessing the cause" to "examining the path."

What We Did
What we built for Hermes is a set of OpenTelemetry (open telemetry frame)-based Tracing Analysis capabilities.

The core idea is straightforward: install runtime instrumentation in the Python environment where Hermes runs, establish spans around the key execution borders of Hermes, and then report traces and indicators to the observability backend through OTLP (OpenTelemetry Protocol), a standard protocol.

Our focus is not on "what the last row of reply looks like", but on the running procedure of Hermes itself.

This Solution Has Several Advantages Worth Highlighting
It is worth mentioning that this set of plugins is not a temporary instrumentation script thrown together, but is designed along the OpenTelemetry system.

First, it follows the GenAI standard specification as closely as possible at the semantics layer. The currently reported trace data preferentially snaps to the OpenTelemetry GenAI semantic conventions. For structures in the Agent runtime that are closer to the execution procedure, extensions are made in combination with LoongSuite Semantic Conventions. Instead of defining a batch of field names that can only be understood internally, we try to use a set of standard, reusable, and portable semantic expressions. In other words, this is not a makeshift approach, but a well-structured observability design that follows industry best practices.

Second, it provides not only traces but also basic metrics signals. In addition to the call chain of a single request, you can also view trends such as the number of invocations, number of faults, invocation duration, and token usage. This way, you can replay a single request along a trace, or observe cost fluctuations, performance changes, and abnormal trends from a global perspective.

Third, it records time to first token (TTFT) separately for streaming scenarios. In many cases, when users perceive something as "slow", it is not necessarily that the entire generation is slow, but rather that the first token takes too long to return. With TTFT, performance issues can be further broken down from "feels slow" into "slow first token" or "slow overall generation".

Fourth, it is not attached to a single Alibaba Cloud service on the backend. The current solution can be directly connected to Alibaba Cloud ARMS, but it uses the OTLP standard protocol underneath and is not designed to be locked into a private data structure. Connecting to ARMS works today, and if you need to connect to other OTLP-compatible backends in the future, migration space is preserved.

Fifth, it supports security audits of important behaviors in Hermes. By collecting full operation logs, access records, and user behavioral data from the Hermes system, and combining outlier detection algorithms to build a dynamic audit model, it can accurately detect suspicious behaviors such as unauthorized access, abnormal data exporting, and malicious prompt injection.

What Can Already Be Seen
The observability capability of the current version of Hermes can revert a real agent run into a ReAct structured trace.

The core pipeline is as follows:

invoke_agent Hermes  
└── react step  
    ├── chat   
   └── execute_tool <tool_name>

If a job contains multiple rounds of inference and multiple tool calls, the pipeline naturally expands:

The significance of this pipeline is not that there are more spans, but that the actual execution of Hermes becomes visible for the first time.

How many rounds an execution ran, which round triggered the tool, and how the tool affected subsequent inference — all of this can now be viewed in the same trace.

Call a Model
Each chat span can currently record:

● gen_ai.request.model

● gen_ai.usage.input_tokens

● gen_ai.usage.output_tokens

● gen_ai.usage.total_tokens

● gen_ai.response.time_to_first_token

This means we can finally view tokens and latency per "actual model invocation" instead of only looking at the aggregate of an entire session. Especially in streaming scenarios, TTFT (time to first token,first-token latency) can help us further distinguish whether the first token is slow to return or the overall generation procedure is slow.

Tool Calling
Each execute_tool span can currently record:

● gen_ai.tool.name

● gen_ai.tool.call.arguments

● gen_ai.tool.call.result

Tools are no longer empty edge zones in the procedure. We can see when Hermes decided to invoke a tool, which tool was invoked, what parameters were passed, and what results were returned.

Agent-Level Summary
The root vertex invoke_agent Hermes span can now record the aggregation results of the entire run, including:

● Cumulative Token

● Final output message

● Total time consumption info

Important Behavior Audit
Records agent behavior across the full chain, intelligently generates audit views, and exposes high-risk operations.

Quick Observability Integration: Deployment in a Few Steps
The integration path for Hermes observability is streamlined into a straightforward flow: get the command from the console, copy it to the terminal and execute it, enable the plugin, start Hermes, and begin reporting.

Tracing Integration
Go to the console to obtain the installation command
Log on to the CMS 2.0 (Cloud Monitor Service 2.0) console, go to the corresponding application monitoring workspace, choose Integration Center > AI Application Observability, and click Hermes.

In the sidebar, enter the application name and click Get to immediately generate the integration command. Click the icon in the upper-right corner to copy it with one click.

One-line command to start installation
Open the terminal on the machine where Hermes is located, paste the copied command, and execute it:

curl -fsSL https://arms-apm-cn-hangzhou-pre.oss-cn-hangzhou.aliyuncs.com/hermes-agent-cms-plugin/hermes-cms.sh | bash -s -- install \  
  --x-arms-license-key "auto" \  
  --x-arms-project "Your project" \  
  --x-cms-workspace "Your Workspace" \  
  --serviceName "hermes" \  
  --endpoint "https://Your ARMS-OTLP address/apm/trace/opentelemetry"

When you execute the installation command for the first time, in addition to installing the plugin itself, the system also registers the hermes-cms command on the local machine for subsequent operations such as enable, disable, and uninstall.

If the following message appears in the terminal, the plugin has been installed successfully:

════════════════════════════════════════════════════

✅ hermes-agent-cms-plugin installed successfully!

════════════════════════════════════════════════════

Throughout the procedure, you do not need to manually edit the configuration file. The script will first match the current environment. Only when the current environment does not meet the requirements will it resume trying the official default installation position.

Turn on observability, and then start Hermes
After the installation is complete, don't rush to check the console.

The first step is to turn on the observability switch:

hermes-cms enable

Then start Hermes.

To run in the foreground, execute directly:

hermes

Run executable in background:

hermes gateway install

hermes gateway start

How to confirm that instrumentation is actually working
If the following tooltip appears in the terminal after startup, the observability instrumentation has taken effect:

loongsuite-site-bootstrap: started successfully (OpenTelemetry auto-instrumentation initialized).

After confirming that the instrumentation has taken effect, send a few test requests to Hermes to run a real job that triggers multiple rounds of inference and tool calling. After a minute or two, return to the CMS 2.0 console, and you will see your Hermes application in AI Application Observability.

At this point, Hermes is no longer just a black box responder — it becomes a running system that can be expanded, tracked, and analyzed.

Enter our observability application to view not only the number of Hermes model invocations, token consumption trends, request fluctuations, and the average number of LLM invocation rounds per request, but also the latency and invocation distribution across AGENT, LLM, and TOOL phases. You can also trace a complete Trace to revert the actual execution procedure of Hermes, clearly seeing how many rounds of inference a job went through, which tools were invoked, which step took the longest, and which round consumed the most tokens.

View the demo examples and the hermes_agentloop_support example at https://sls.aliyun.com/doc/en/playground/cmsdemo.html

Want to shut down or uninstall? It's straightforward.
To temporarily shut down observability, execute:

hermes-cms disable

To completely uninstall the plugin, execute:

hermes-cms uninstall

Log Ingestion
Configure application info on the access Card
Next, click the "Log Access" page, set a custom application name, click Initialize Resources, enter the previously configured Project name, and configure the machine group as prompted to complete the Hermes Audit Feature with one click.

Auto-generated Audit dashboard
After the access is complete, in the left sidebar, choose Audit > Hermes Insight > Hermes Audit to view the audit dashboard of your Hermes agent.

Summary and Outlook
This solution can reliably address Tracing Analysis, token attribution, and basic performance breakdown, while also providing basic metrics signals for trend analysis. However, this does not mean that all observability work for Hermes is complete.

Next, we will continue to push forward in several directions.

● On the data plane, continue to expand from traces, span properties, and basic indicators to more complete log audit and runtime diagnostics capabilities.

● On the link plane, continue to refine Hermes-specific execution phases beyond agent, step, llm, and tool, such as memory lifecycle, delegation orchestration, and runtime recovery.

● On the governance plane, continue to strengthen content collection control, finer-grained data governance capabilities, and unified desensitization and security policy development.

Today, we already have an active runtime observability infrastructure, and the next goal is to further evolve it into a more complete, more detailed Agent observability system that is better suited for real production environments.

From Observable to Understandable: Building Agent-Native Code Knowledge Graphs with UModel

ObservabilityGuy — Mon, 11 May 2026 06:57:40 +0000

UModel builds agent-native code knowledge graphs using deterministic AST parsing and cross-domain associations for deeper AI code understanding.

Background
In recent years, AI agents (Cursor, Copilot, Claude Code, Codex, etc.) have become deeply involved in software development. From code completion to cross-file refactoring, from bug localization to architecture design, agent capabilities are growing stronger. From Prompt Engineering to Context Engineering to Harness Engineering, the ways to harness AI continue to evolve, and the capability boundaries of agents continue to expand.

However, when we hand a real enterprise-level project to an agent, an overlooked question begins to surface: Does the agent really understand your project?

The way agents currently understand code is diverging into two distinct schools:

● No-index school: Claude Code follows the Unix philosophy and performs no pre-indexing at all — it searches the file system in real time using grep, rg, and glob. Anthropic's internal tests found that agentic search outperforms retrieval-augmented generation across the board, by a lot. It is concise, real-time, and free of privacy issues, but each session starts from scratch and is costly for large repositories.

● CodeIndex School: Cursor, Windsurf, and Copilot follow the vector index route: using tree-sitter for semantic text segmentation, generating embeddings and storing them in a vector database (such as Turbopuffer), then using Merkle tree for incremental synchronization. Qodo and Augment Code go a step further by overlaying a code dependency graph and commit history index on top of the vector index.

Both schools have their own strengths, but they still struggle with the following problems:

● I want to change the Adapter interface of pkg/a2a. What is the scope of impact?

Vector similarity search cannot find the dependency chain, and grep-based file-by-file search is inefficient and incomplete.
● In production, the vibeops-xxx SLO has been breached with a large number of pending requests. What is the cause? Is it a code change?

The code index only covers the code domain; O&M domain data is not in the graph.
● Are there any abnormal dependencies in the project that cross architecture borders?

Without architecture level modeling, crossing borders cannot be defined.
What these problems have in common is that they require deterministic structural relationships, cross-domain entity associations, and change history across the time dimension.

The author has been working in the observable field for more than ten years, reviewing the development of observable, especially with the increasing complexity of cloud native and AI native systems, observable has long faced not only "looking at a log and staring at a monitoring chart", but also putting the scattered objects such as applications, services, containers, databases, alarms, changes and events back into the same context, answer "who is related to whom", "how the impact is spread" and "when did the problem begin to occur".

Because of this, Alibaba Cloud can observe the gradual evolution from the collection and display of scattered data such as logs, indicators, and links to the unified modeling of object-oriented, relationship, and time series. UModel is precipitated under this practical background.

This is strikingly similar to the trajectory of the observability realm: from viewing logs to unified modeling, observability evolved from fragmented data to the UModel knowledge graph. Yet code understanding, even with the most advanced CodeIndex solution, remains at the stage of helping agents find relevant snippets — the snippets are found, but the structure is not understood.

Five Paradigms of Code Understanding
Before diving into the technical solution, it is necessary to clarify the complete landscape of current code understanding. The five paradigms represent the evolution from stateless search to stateful inference.

Paradigm 1: Agentic Search (Claude Code School)
Claude Code is currently the most extreme index-free route. Anthropic founding engineer Boris Cherny publicly shared the story behind this decision: early versions of Claude Code used retrieval-augmented generation + a local vector library, but internal tests found that agentic search won comprehensively — by a lot, and this was surprising.

Its approach is pure to the point of elegance:

Agent receives a question  
  → Glob: pattern matching by file name (near-zero token cost)  
  → Grep (ripgrep): regex search by content (low token cost)  
  → Read: read the complete file (high token cost)  
  → Evaluate → next round of search or provide an answer

Tools are tiered by token cost, and the agent independently determines the search policy — like an experienced developer using rg + cat in the terminal to troubleshoot issues. This Unix-philosophy method has several real advantages:

● Zero pre-processing: no index build time required — open the project and start working immediately

● Always Fresh: No index expiration issues. Every search reflects the real-time file system status.

● Privacy-Friendly: Code never leaves your local machine — no embeddings are generated, and nothing is uploaded to any server.

● Simple and Reliable: The dependency chain is extremely short: Agent + file system + ripgrep. No vector database to crash.

But the ceiling of this approach is equally clear:

● No Structure Awareness: rg HandleRequest can find all occurrences, but cannot distinguish definitions from invocations or comments. The Agent has to read the code itself to determine this.

● Start from Scratch Every Time: Dependencies analyzed in the previous session are entirely discarded in the next. There is no persistence of accumulated knowledge.

● Limited scale: A TypeScript project with 200 files is fine, but for an enterprise-level monorepo with 50,000 files, agentic search may require 30+ rounds of tool calling and tens of thousands of tokens to piece together a global dependency graph. In practice, it is impossible to construct a complete global graph — only partial views relevant to the current job can be assembled.

● Unable to perform global analysis: Cannot answer "list all invocations across architecture levels" because the architecture levels themselves have not been modeled.

Paradigm 2: CodeIndex / Vector Index (Cursor, Windsurf, and Copilot School)
This is the mainstream technical approach of current AI IDEs. Taking Cursor as an example, its technical architecture has been extensively analyzed in public:

Code Repository  
  → Parse into AST with tree-sitter  
  → Segment by semantic unit (function, class, logic block)  
  → Generate vector embedding  
  → Store in Turbopuffer vector database  
  → Merkle Tree tracks changes for incremental synchronization

Cursor has achieved several elegant optimizations in engineering: it uses Merkle Tree root hash comparison to detect changes every 10 minutes and only re-embeds changed files; 92% codebase similarity among team members allows index reuse, reducing the initial indexing for new members from minutes to seconds; the index scope is controlled via .cursorignore.

Windsurf (Codeium) uses a similar retrieval-augmented generation architecture: 768-dimensional vector embedding + proprietary M-Query retrieval, but additionally overlays the Cascade context engine to track edit history, terminal commands, navigation patterns, and other session states. GitHub Copilot achieved sub-second semantic search indexing in March 2025.

The real value of CodeIndex is semantic search: the agent can find relevant code by describing intent in natural language without knowing the exact function name. This is something grep cannot do.

But CodeIndex has a fundamental limitation: vector similarity is text-level approximate matching, not structure-level relational reasoning.

● import pkg/a2a is a deterministic dependency in code, but in vector space it is merely a similarity signal of a text segment.

● Finding all modules that directly or indirectly depend on pkg/a2a requires graph traversal, not AISearch.

● Determining how many hops the impact of this interface change propagates along the invocation chain requires deterministic call relationships, not semantic similarity.

● Augment Code's evaluation shows that Cursor produces inconsistencies in cross-file refactoring across 50+ files: the first 30 files are modified correctly, but the last 20 contain faults due to context window overflow.

CodeIndex is essentially a smarter search engine: it helps agents find the correct snippets to insert into the context, but does not perform structured inference for agents.

Paradigm 3: Code Graph + Retrieval-Augmented Generation Hybrid (Qodo and Augment Code School)
Qodo and Augment Code represent the next evolutionary direction of CodeIndex: layering code structure graphs on top of vector indexes.

Qodo's technology stack is particularly rigorous:

● Self-developed Qodo-Embed-1 code embedding model (1.5B parameters surpassing 7B competitors on the CoIR benchmark), capturing syntax, variable dependencies, control flow, API usage, and other code-specific semantics through synthetic data training

● Client-side code graph building: functions, classes, modules and their call graphs, inheritance relationships, and cross-language links

● Server-side maintenance of vector database + design documents + architecture diagrams + PR/commit history

● AST-aware segment policy: recursively chunk AST edge zones and backfill key contexts such as import statements and class definitions

Augment Code 's Context Engine goes even further:

● Semantic index across repositories to understand how services connect and depend on each other

● Index beyond Code: commit history (why changes were made), codebase patterns, external documents, tickets, and even tribal knowledge

● Released Context Lineage in 2025 to index commit histories and diff summaries, enabling agents to understand the evolution of architectural decisions

● Open to any compatible agent via MCP protocol, with benchmarks showing 30–80% quality improvement

The key advancement of this school of thought is that code is not just text, but a structured graph. Augment, in particular, demonstrates the insight that understanding requires context, and context requires history.

However, even the most advanced code graph + retrieval-augmented generation hybrid solution still has several systemic borders:

● The graph scope is limited to the code domain: It knows that A invokes B, but not what alerts the service corresponding to B has triggered in the production environment. The code graph and the O&M graph are disconnected.

● Limited graph query capabilities: Graphs serving retrieval-augmented generation typically support neighbor lookup and short-path queries, but do not support arbitrary-depth graph traversal, pattern matching, or aggregation and analysis.

● IDE-local, not team-global: The index is attached to a developer's IDE instance. Structural insights analyzed by one person cannot be directly reused by another.

● Lack of a standardized timing dimension: Augment's Context Lineage has started incorporating commit history, but build logs, deployment logs, test logs, and event logs — these complete temporal memories are not yet in the graph.

Paradigm 4: CodeWiki / LLM Document (DeepWiki School)
DeepWiki (GitHub 15.7k stars, produced by the team behind Cognition AI / Devin) represents another approach: Code Repository → LLM → polished Wiki document. Simply replace github.com in the URL with deepwiki.com to see the automatically generated architecture diagrams, module documents, and function annotations.

This provides an excellent experience for developers to quickly understand unfamiliar projects. DeepWiki also supports controlling the generation scope through the .devin/wiki.json configuration file, and provides tool interfaces such as ask_question, read_wiki_structure, and read_wiki_contents via the MCP Server.

But documents are essentially linear narratives optimized for human reading:

● Hard to authenticate: Descriptions generated by LLMs may hallucinate, and in code understanding, an incorrect "A invokes B" is more dangerous than no information at all.

● Hard to traverse: Documents cannot answer graph traversal queries such as "list all functions that invoke X."

● Difficult to infer: Multi-hop analysis is not supported: if A is changed, following the calls relationship for 3 hops, which entry points are affected?

● Difficult to maintain: Changing a single line of code requires full regeneration. Although DeepWiki supports badge-triggered auto-refresh, each time it invokes a full LLM call, resulting in high cost and latency.

● Not programmable: The MCP interface essentially asks a document a question, rather than executing a query on the graph.

The relationship between CodeWiki and CodeIndex is similar to the relationship between materialized views and DPI engines in the database realm: documents are precomputed views that answer preset questions quickly, but cannot answer ad-hoc queries outside the view.

Paradigm 5: Code Knowledge Graph (Our Choice)
The five paradigms can be arranged along a single axis: from "stateless search" to "stateful inference".

If Agentic Search is each on-site survey, CodeIndex is surveying with a high-definition map, Code Graph + retrieval-augmented generation is a map annotated with highways and railways, and CodeWiki is a commissioned local chronicle: then what we want to build is a living GIS system: you can query the path between any two points, overlay real-time traffic data, annotate the traffic history of each road, continuously update as the terrain changes, and support storage analysis in any dimension.

The key difference is not better search, but a systematic combination of three dimensions:

1.Deterministic vs. Probabilistic: CodeIndex gives you the most likely relevant snippets (vector similarity). Code Graph gives you structural relationships parsed from the AST (but query capability is limited by the retrieval-augmented generation frame). We give you deterministic AST fetch + SPL/graph-match arbitrary query: confidence level 1.0 relationships + a Turing-complete query language.

2.Code domain vs cross-domain: From Agentic Search to Code Graph + retrieval-augmented generation, all solutions stop at the code domain. Which functions does this module invoke: answerable. How many alerts did the production service corresponding to this module have last week: unanswerable. UModel's EntitySetLink can connect code.module to ops.service, event.alert, and req.issue. The agent infers along the link without needing to jump out of the graph.

3.Snapshot vs timeline: CodeIndex is a snapshot index of the current code. Code Graph is starting to incorporate commit history. We provide a complete time dimension: commit_log, build_log, deploy_log, test_log, and incident_log. Each LogSet is associated with an EntitySet through DataLink. The agent not only knows what the current structure is, but also how it evolved to this point and how it performs in production.

From Personal Wiki to Code Wiki: One Paradigm, Different Certainty

The personal Wiki flow is: source data → LLM extracts entities and relationships → snap and normalization → UModel structure layer → Wiki pages. The entire extraction procedure depends entirely on the LLM, so each relationship is inherently uncertain: Are Zhang Cheng and Yuan Yi the same person? Is this article related to that project? Both require LLM judgment and correction by the snap layer.

There is one fundamental difference in the code realm: the structural relationships of code are deterministic.

import pkg/a2a imports pkg/a2a, and func (s *Server) HandleRequest() is a method of the Server class: these do not require LLM inference — AST parsing can determine them with a confidence level of 1.0.

This means that code wikis can introduce a model layer deterministic guarantee on top of the personal wiki paradigm:

Personal Wiki:   Source material → [LLM fetch] → Snap → UModel → Wiki Page  
                          ↑ Entirely dependent on LLM, confidence level 0.4–0.9  

Code Wiki:   Code Repository → [AST deterministic fetch] + [LLM semantics enhancement] → UModel → CLI query  
                          ↑ Structural relationships determined (1.0)   ↑ Summary/attribution supplement (0.6–0.9)

This layer of determinism is critical to the agent's reasoning: when the agent performs RCA, it needs to trust every hop on the invocation chain. If a calls relationship is guessed by the LLM, the entire reasoning chain becomes unreliable. Relationships fetched by AST are deterministic facts that the agent can trust unconditionally.

At the same time, the code wiki retains the LLM enhancement capabilities of the personal wiki: semantic layer information such as module summaries, document-code associations, and widget attributions is still generated by the LLM, annotated as INFERRED, and the agent can selectively accept it.

Entity + Log + Link: Not Just a Structure Graph
The core design of UModel in the observability realm is to describe the IT world with a graph composed of sets and links: EntitySet describes the current state of entities, LogSet describes timing management events, MetricSet describes measure indicators, and Link connects them into a network.

When we apply the same modeling methodology to the code realm, we get more than just a structure graph.

Entity: Current Code Structure
Five types of EntitySets describe the current state of the code and support the coexistence of multiple repositories through repo_id composite primary keys:

repo_id participates in the primary key calculation (Entity ID = md5(repo_id:pk_value)), so that modules with the same name in different repositories do not conflict, and a single graph can accommodate multiple projects simultaneously.

Six types of EntitySetLink describe structural relationships: contains, imports, calls, extends, describes, and belongs_to. Each relationship is annotated with confidence and extraction_method (EXTRACTED / INFERRED / AMBIGUOUS).

Log: The Change History of Code
This is a critical watershed between Code-WIKI and all pure graph tools.

In the observability realm, we look at not only the current status of a pod (Entity), but also its logs and metric trends. Code is the same: looking only at the structure without the history is like looking at a single screenshot.

Logs in the code realm go far beyond Git commits:

The value of logs lies in the associated query with entities:

● Who modified this module in the last week? →commit_log WHERE module_path = X AND time > now()-7d

● Have any new incidents occurred since the last deployment? →deploy_log JOIN incident_log ON time_window

● Has the build time increased after introducing this dependency? →build_log GROUP BY week, cross-referencing dependency change time in commit_log

Each LogSet is associated with the corresponding EntitySet through DataLink. The agent can navigate from an entity to a log, or trace back from a log to an entity.

Cross-Domain Association: Code Is Not an Island
Code never exists in isolation. It serves requirements, reaches production through CICD, generates observable data at runtime, and traces back to the code for troubleshooting when issues arise. In the current toolchain, each link is an island: requirements are in Jira, code is in Git, builds are in Jenkins, services run in K8s, and alerts are in the monitoring system.

When a production alert fires, how many systems must you jump through and how many pieces of info must you manually correlate to trace from the alert back to the code change?

The value of UModel is that all these entities can live in the same graph.

Technical Architecture: Dual-Track Fetch + Graph Build
Overall Pipeline

DETECT: Incremental Change Detection
A SHA256 content fingerprint is computed for each file and compared against the cache from the last build. For vibeops-agents (~2,375 Go files), an incremental build typically processes only dozens of changed files, reducing the time from minutes to seconds.

EXTRACT: AST + LLM Dual Track
AST track (tree-sitter): A PEG-based incremental resolver that supports 40+ languages. It uses tags.scm rules to consistently fetch definitions, references, structural relationships, import relationships, invocation relationships, and inheritance relationships across languages. All extraction results have a confidence level of 1.0.

Notably, CodeIndex solutions such as Cursor also use tree-sitter. However, they use tree-sitter for semantic text segmentation (splitting code into chunks suitable for embedding), whereas we use tree-sitter for structure extraction (fetching deterministic relationships such as definitions, references, invocations, and inheritance). The same resolver serves completely different goals: the former produces vectors, and the latter produces a graph.

LLM track: Module summaries (agent context injection segments, not human-readable documents), document-code associations, and widget attribution. Each is annotated with extraction_method: INFERRED + confidence level. Agents can select a trust threshold by scenario: RCA prefers high confidence levels, while exploration scenarios can be relaxed.

RESOLVE: Cross-file Symbol Parsing
Single-file AST cannot resolve cross-file references. RESOLVE handles the following:

● Go import github.com/org/repo/pkg/a2a→ module_path pkg/a2a

● Method receiver type (s *Server)→ attribution code.type pkg/server.Server

● Invoke s.HandleRequest()→pkg/server.Server.HandleRequest

● Interface implementation type Adapter struct implements Handler→ extends relationship

Deterministic parsing, no dependency on LLM.

BUILD: Graph Assembly + Architecture Discovery
Architecture discovery is not simple community detection: Louvain/Leiden discovers clusters, not architectures. Complete flow:

Step 1: Graph construction  
  Modules as edge zones, imports + calls + extends as directed edges  
  Edge weight: calls > imports > extends  

Step 2: Hierarchical analysis  
  Compute dependency directionality: A→B and B↛A → A is above B  
  Detect top-level entries with indegree = 0 and underlying infrastructure with outdegree = 0  

Step 3: Community detection  
  Leiden algorithm discovers functional clusters on directed graphs  
  Resolution parameter controls granularity (~150 modules → ~15 widgets)  

Step 4: Annotation and naming  
  Annotate hierarchy based on dependency direction: API/Gateway, Service/Business, Infrastructure/Utility  
  LLM naming and description, cross-validation with project documents

The output is a hierarchical, directional, named architecture view. The agent can use this to determine whether an invocation crosses architecture layers.

SYNC: Synchronize to UModel

Entity write: starops umodel post-logs → __entity logstore  
Topo write:  starops umodel post-logs → __topo logstore  
Schema synchronization: starops umodel sync (register EntitySet/Link definitions)

The UModel backend is based on the Simple Log Service storage engine and inherits capabilities such as high-throughput writes, second-level query, graph-match graph traversal, SQL aggregation, and full-text index.

SERVE: Engineering Details of the Query
Key patterns explored in practice:

Two-step query: graph-match returns entity_id without business fields. All graph traversal queries first traverse the topology to obtain the ID set, then pull business fields in batches:

Step 1: .topo | graph-match (n1:code@code.module {__entity_id__: '<id>'})  
              -[e]->(n2) project n1, e, n2  

Step 2: .entity with(domain='code', name='code.module', ids=['id1','id2',...])

Aggregation via direct Simple Log Service (SLS) query: Statistical queries such as hot spot analysis directly run SQL against the __topo Logstore:

SELECT dest_entity_id, count(1) as import_count

FROM log WHERE relation_type = 'imports'

GROUP BY dest_entity_id

ORDER BY import_count DESC LIMIT 20

At the current multi-repository scale (~11,000 entities, ~19,000 edges, including the vibeops-agents and starops-cli projects), the end-to-end latency of a single query is in the hundreds of milliseconds.

Agent Interaction Layer: Command-Line Interface (CLI) + Skill
CLI Design
The agent's reasoning is progressive: search first, see the results, and then decide the next step. The CLI's search→context→impact naturally matches this pattern and supports batch execution and MPS queue combinations.

code-wiki query <subcommand>     # graph query  
  ├── search <keyword>       # entity search  
  ├── context <name>         # full context of a symbol  
  ├── impact <path>          # change impact analysis  
  ├── callers / callees      # invocation chain  
  ├── deps / rdeps           # dependencies / reverse dependencies  

code-wiki check <subcommand>     # administration check  
  ├── arch                   # architecture violation scan  
  └── hotspots               # coupling hot spots  

code-wiki ingest             # build/update graph  
code-wiki status             # health check

Subcommands are organized by agent intent. The agent does not need to know whether the underlying implementation is graph-match or Simple Log Service SQL: use impact to view the impact scope.

Output Format: Optimized for the Agent Context Window
The default --format brief output is optimized for the agent's token budget:

$ code-wiki query context pkg/a2a  

Module: pkg/a2a  
  LOC: 1,247 | Language: Go | Component: a2a-protocol  
  Summary: A2A protocol implementation for agent-to-agent communication  

Types (17): TaskStore(struct), A2AServer(struct), AgentCard(struct), ...  
Functions (52): HandleA2ARequest[entry], StartA2AServer[entry], ...  
Reverse dependencies (9): pkg/api/handler, pkg/server, cmd/vibeops-agents, ...  
Component crossings: → api, → scheduler

The output of a query context is < 500 tokens. Use --format json when full data is required.

Skill: Scenario-based User Guide
Agent Skills with the command-line interface (CLI) are organized by scenario. Agents do not need to learn Structured Process Language syntax:

## RCA: From alerting to code  
code-wiki query search <keyword>       # Locate module  
code-wiki query context <module>      # Understand structure  
code-wiki query callers <function>    # Trace invocation chain  

## Development: Evaluate impact before changing code  
code-wiki query impact <module>       # Impact scope  
code-wiki query deps / rdeps          # Dependencies  

## Administration: Regular checks  
code-wiki check arch                  # Architecture violations  
code-wiki check hotspots              # Hot spot analysis

Case Study
Case 1: Agent Independently Completes Change Impact Assessment
We assign a job to a sub-agent:

Assess the impact scope of modifying the pkg/a2a module in the vibeops-agents project, including affected modules and entry points, upstream and downstream dependencies, and architecture threats.

The sub-agent completed the following inference only by code-wiki** command-line interface (CLI) and Skill**, without reading any source code files:

$ code-wiki query search a2a  
Found 9 entities matching a2a:  
  [module] pkg/a2a                    LOC:1247  A2A protocol core  
  [module] pkg/a2a/adapter            LOC:834   A2A adapter implementations  
  [module] pkg/a2a/taskstore          LOC:567   A2A task state management  
  ...  

$ code-wiki query context pkg/a2a  
Module: pkg/a2a  
  Types (17): TaskStore, A2AServer, AgentCard, ...  
  Functions (52): HandleA2ARequest[entry], StartA2AServer[entry], ...  
  Reverse dependencies (9): pkg/api/handler, pkg/server, cmd/vibeops-agents, ...  

$ code-wiki query impact pkg/a2a  
Impact of pkg/a2a:  
  Direct dependents (9): pkg/api/handler, pkg/server, cmd/vibeops-agents, ...  
  Entry points affected: HandleA2ARequest, StartA2AServer, main  
  Component crossings: a2a-protocol → api, a2a-protocol → server  

$ code-wiki check hotspots  
Top hotspots by import count:  
  1. pkg/a2a/adapter    (48 imports)  ← High coupling warning  
  2. pkg/util/logger     (35 imports)  
  3. pkg/config          (32 imports)

Agent final output:

=== A2A Protocol Modification Impact Assessment ===  
  
Scope of Impact:  
- Directly dependent modules: 9 (pkg/api/handler, pkg/server, etc.)  
- Entry points affected: 3 (HandleA2ARequest, StartA2AServer, main)  
- Cross-component borders: 2 (a2a-protocol → api, a2a-protocol → server)  
  
Threat points:  
- pkg/a2a/adapter is imported by 48 modules, the highest coupling degree in the entire project  
- bus factor = 1, consider adding a reviewer  
  
Suggested execution order:  
1. Modify the pkg/a2a core protocol definition first  
2. Synchronize the interface implementation of pkg/a2a/adapter  
3. Modify the invocation method of pkg/api/handler  
4. Update the initialization logic of pkg/server  
5. Regression testing on the cmd/vibeops-agents entry point

5 commands in total, each taking 1–3 seconds, with a total time of less than 15 seconds. The Agent did not read any source code files and completed a full impact assessment solely through graph queries.

Case 2: RCA: From Alerting to Code Root Cause
Production alerting: service-vibeops error_rate > 5%

# 1. Locate the code module from the O&M entity  
$ code-wiki query context pkg/server  
Module: pkg/server  
  Functions (23): StartServer[entry], handleRequest, applyMiddleware, ...  
  Dependencies (12): pkg/a2a, pkg/config, pkg/auth, ...  
  
# 2. Trace the invocation chain to locate the potentially faulty downstream  
$ code-wiki query callees pkg/server.handleRequest  
Callees of handleRequest:  
  pkg/auth.ValidateToken          [component: auth]  
  pkg/a2a.HandleA2ARequest        [component: a2a-protocol]  
  pkg/scheduler.DispatchTask      [component: scheduler]  
  
# 3. Check commit_log and find that the a2a module was changed 2 hours ago  
#    author=xxx, message=refactor adapter interface  
  
# 4. Confirm the impact of the change  
$ code-wiki query impact pkg/a2a  
Impact of pkg/a2a:  
  Direct dependents (9): pkg/api/handler, pkg/server, ...  
  Entry points affected: HandleA2ARequest, StartA2AServer  
  
# → Root cause: The a2a interface refactoring affected the server invocation chain. Check interface compatibility.

Case 3: Architecture Administration: Detecting Architecture Decay

# 1. Scan for architecture violations  
$ code-wiki check arch  
Architecture violations:  
  pkg/util/logger calls pkg/api/handler.GetRequestID  
    [utility → api] The utility layer should not invoke the api layer  
  pkg/config calls pkg/scheduler.GetDefaultConfig  
    [infra → service] The infrastructure layer should not depend on the business layer  
  
# 2. Identify coupling hot spots  
$ code-wiki check hotspots  
Top hotspots:  
  1. pkg/a2a/adapter      48 imports  [HIGH]  
  2. pkg/util/logger       35 imports  [NORMAL]  
  3. pkg/scheduler/queue   28 imports  [MEDIUM]  
  
# 3. Analyze the highly coupled module in depth  
$ code-wiki query rdeps pkg/a2a/adapter  
Reverse dependencies (48):  
  pkg/api/* (12 modules), pkg/server/* (8 modules), pkg/scheduler/* (6 modules), ...  
  
# Agent suggests splitting into adapter/protocol, adapter/transform, and adapter/routing

Outlook
Comprehensive Digital Evaluation
We plan to build a standardized code comprehension evaluation benchmark covering core scenarios such as impact analysis, invocation chain tracing, architecture violation detection, and RCA root cause localization. On real codebases of varying scales, we will compare the performance of three paradigms — Model + Bash (Agentic Search), Model + CodeWiki (LLM document), and Model + UModel (knowledge graph) — across dimensions including accuracy, recall rate, number of inference steps, and token consumption.

Use SWE-bench-style quantization evaluation to make the capability borders of each paradigm measurable and reproducible. Based on this, optimize the overall technical architecture based on benchmark fractions, including iterative upgrades to related skills and the command-line interface (CLI).

Agent Self-Maintenance
Agents are not just graph consumers, they can also be maintainers:

● After a code schema evolution, the associated LLM-inferred relationships are marked for reevaluation

● Regularly inspect orphaned entities, missing relationships, and expired data

● On top of the above capabilities, a verification and quality assessment system is also needed to make self-maintenance controllable.

Architecture Guard Gate
Integrated into the CI flow, automatically run on PR:

codecode-wiki ingest --incremental        # Incremental graph update  
code-wiki check arch                  # Architecture violation check  
code-wiki query impact <changed_files> # Change impact analysis

From Observable to Understandable
From modeling observable data to modeling code knowledge, from describing running systems with Entity + Log to describing code systems with Entity + Log: UModel is evolving from observing IT systems to understanding the code and procedures that build them.

When agents truly understand the structure, history, and production performance of code simultaneously, genuinely AI-native software engineering becomes possible.

Build Alibaba Cloud API Gateway Monitoring with Realtime Compute for Apache Flink and SLS

ObservabilityGuy — Mon, 11 May 2026 03:23:05 +0000

This article introduces how to build a real-time, scalable API gateway monitoring system for Alibaba Cloud Open Platform using Realtime Compute for Apache Flink and SLS.

By Pan Weilong (Alibaba Cloud Observability), Ruan Xiaozhen (Alibaba Cloud Open Platform)

Background and Challenges
Background

Alibaba Cloud Open Platform is the standard entry point for developers to manage cloud resources. The Open Platform hosts the external APIs of almost all cloud products, and allows for automated O&M and cloud resource management. As enterprise dependency on automation deepens, the stability of the Open Platform becomes crucial.

The stakeholders of the monitoring system include:

● Open Platform's O&M team: Responsible for the overall availability of the API gateway, requiring centralized monitoring and alerting capabilities.

● Cloud product teams (such as ECS, RDS, and SLB): Need to view the API call metrics and dashboards of their own products, and configure fine-grained alerting.

● SRE teams: Need to quickly locate faults and perform root cause analysis.

Fluctuations in any API may impact the production business of customers. Therefore, a comprehensive metric monitoring system must be established, accompanied by timely alerting capabilities to ensure high availability.

Challenges
The primary data source for the monitoring system is the access logs of the API gateway. These logs are generated by gateway nodes distributed across various regions. The system faces the following challenges:

Solution
To address those challenges, we adopt the cloud-native combination of Realtime Compute for ApacheFlink and SLS to build a real-time monitoring system.

Components
The core components of this solution and the adoption rationale are as follows:

The advantages of this solution are:

● Fully managed: SLS and Realtime Compute for Apache Flink are both fully managed services, eliminating the need to manage infrastructure.

● Scalability: Consumption throughput and compute resources can be scaled on demand.

● End-to-end guarantee: End-to-end observability, from collection to alerting.

Architecture

The entire data processing pipeline adopts a regional deployment and centralized aggregation design. Log collection and aggregation are completed within each region to reduce latency. Processed metric data is aggregated cross-region to a single MetricStore for centralized monitoring.

Intra-region Processing
An independent data processing pipeline is deployed in each region to reduce latency:

1.Data collection: Logtail collects the gateway node logs in real time. Logtail is a high-performance, proprietary log collector from Alibaba Cloud. It has the capabilities of millisecond-level latency and a throughput of millions of EPS, ensuring the reliable transmission of massive logs.

2.Log storage: The SLS Logstore stores the raw API access logs in the region. It supports real-time query and analysis of request details, and serves as the data source for Flink stream processing.

3.Regional aggregation: Flink Job 1 is independently deployed in each region. It's joined with MySQL dimension tables (storing metadata, such as the cluster information of gateway nodes and API business domains like ECS) to aggregate business metrics. This can significantly reduce the size of data for cross-region transmission.

Cross-region Aggregation
Local aggregation results are sent to a single MetricStore:

4.Cross-region aggregation: Flink Job 2 (metric transform) is independently deployed in each region, adding timestamp info to the aggregation results, and aggregating the results to the centralizedSLS MetricStore. This allows the O&M team to view the metrics of all regions centrally.

5.Visualization and alerting: Connect Grafana to the centralized SLS MetricStore, and query multi-dimensional metrics using standard Prometheus Query Language (PromQL), and alert on abnormal metrics.

Layered Design
The layered design effectively balances data freshness and resource efficiency:

Why not one-layer aggregation?

Avoid data skew: The API traffic distribution is extremely uneven, and the QPS of certain products (such as ECS) is thousands of times that of other products. Grouping data by product will cause data skew and state bloat in specific Flink tasks.
Improve resource efficiency: Regional aggregation reduces data sent downstream by more than 90%, which significantly lowers compute and storage overhead. Metric System Design The target metric system is composed of metrics and labels, covering the following four dimensions:

Metric naming pattern: Prefix_MetricName. For example, the QPS metric of ECS is namespace_product_gw_http_req.

Flink Job Development
Job 1: Intra-region Processing
Consumes raw logs, joins with MySQL sources, and performs two-stage aggregation: fine-grained multi-dimensional aggregation (by product, API, tenant, etc), followed by global metric aggregation.

1.Data Source: Raw logs
Logtail collects raw logs from gateway nodes. Sample log:

{  
  "AK": "STS.NZD***Lgwc",  
  "Api": "DescribeCustomResourceDetail",  
  "CallerUid": "109837***3503",  
  "ClientIp": "192.168.xx.xx",  
  "Domain": "acc-vpc.cn-huhehaote.aliyuncs.com",  
  "ErrorCode": "ResourceNotFound",  
  "Ext5": "{\"logRegionId\":\"cn-huhehaote\",\"appGroup\":\"pop-region-cn-huhehaote\",\"callerInfo\":{...},\"headers\":{...}}",  
  "HttpCode": "404",  
  "LocalIp": "11.197.xxx.xxx",  
  "Product": "acc",  
  "RegionId": "cn-huhehaote",  
  "RequestContent": "RegionId=cn-huhehaote;Action=DescribeCustomResourceDetail;Version=2024-04-02;...",  
  "TotalUsedTime": "14",  
  "Version": "2024-04-02",  
  "__time__": "1768484243"  
}

Note: Ext5 contains a nested JSON structure (such as caller information and request headers), and RequestContent is request parameters in key-value format. These complex structures need to be parsed.

Based on the log structure, define a Flink source table:

CREATE TABLE openapi_log_source (  
  `__time__` BIGINT,  
  LocalIp STRING,           -- Gateway node IP  
  Product STRING,           -- Product code  
  Api STRING,               -- API   
  Version STRING,           -- API version   
  Domain STRING,            -- Access domain   
  AK STRING,                -- Access Key  
  CallerUid STRING,         -- Caller UID  
  HttpCode STRING,          -- HTTP code   
  ErrorCode STRING,         -- Error code   
  TotalUsedTime BIGINT,     -- Request time in ms  
  ClientIp STRING,          -- Client IP  
  RegionId STRING,          -- Region ID   
  Ext5 STRING,              -- Extended field (nested JSON)  
  RequestContent STRING,    -- Request parameters (k/v format)   
  ts AS TO_TIMESTAMP_LTZ(`__time__` * 1000, 3),  
  WATERMARK FOR ts AS ts - INTERVAL '5' SECOND  
) WITH (  
  'connector' = 'sls',  
  'project' = '*****',  
  'logstore' = 'pop_rpc_trace_log',  
  'endpoint' = 'cn-shanghai-intranet.log.aliyuncs.com'  
);

Watermark strategy: A ts - INTERVAL '5' SECOND watermark allows for up to 5 seconds of out-of-order data. Adjust this value based on your business needs. In production, with Logtail collecting gateway logs, the end-to-end latency is typically 2 to 3 seconds, making a 5-second delay sufficient for most cases. For cross-region scenarios, consider relaxing this to 10 to 15 seconds.

2.MySQL Lookup Source: Metadata Enrichment
To add labels (such as app_group and gc_level) to metrics, associate a MySQL lookup source:

-- Gateway cluster info (join on LocalIp)  
CREATE TABLE gateway_cluster_dim (  
  local_ip STRING,  
  app_group STRING,          -- Cluster name   
  region_id STRING,          -- Region ID  
  PRIMARY KEY (local_ip) NOT ENFORCED  
) WITH ('connector' = 'jdbc', ...);  

-- Tenant info (join on Uid)  
CREATE TABLE user_level_dim (  
  uid STRING,  
  gc_level STRING,           -- Customer level (GC5/GC6/GC7)  
  PRIMARY KEY (uid) NOT ENFORCED  
) WITH (  
  'connector' = 'jdbc',  
  'url' = 'jdbc:mysql://xxx:3306/dim_db',  
  'table-name' = 'user_level',  
  'lookup.cache.max-rows' = '50000',       -- Max num of rows to cache  
  'lookup.cache.ttl' = '10min',            -- Cache TTL  
  'lookup.max-retries' = '3'               -- Max retries   
);

Cache policy: In production, gateway_cluster_dim adopts the ALL policy: loads data upon startup and refreshes regularly. user_level_dim uses the LRU policy: caches 50,000 hot spot tenant data records and sets the TTL to 10 minutes to balance the hit rate and data freshness.

3.Job 1 Output: Write to Regional Aggregation Log
The processing results are written to the SLS Logstore machine_agg_log as intermediate storage.

-- Define a regional log aggregation sink  
CREATE TABLE machine_agg_log_sink (  
  window_start TIMESTAMP(3),  
  product STRING,  
  api STRING,  
  version STRING,  
  caller_uid STRING,  
  region_id STRING,  
  app_group STRING,  
  gc_level STRING,  
  http_code STRING,  
  error_code STRING,  
  qps BIGINT,  
  rt_mean DOUBLE,  
  slow1s_count BIGINT,  
  http_2xx BIGINT,  
  http_5xx BIGINT,  
  http_503 BIGINT  
) WITH (  
  'connector' = 'sls',  
  'project' = '****',  
  'logstore' = 'machine_agg_log',  -- Logstore name  
  'endpoint' = 'cn-shanghai-intranet.log.aliyuncs.com' -- Replace it with actual endpoint   
);  

-- Insert data  
INSERT INTO machine_agg_log_sink  
SELECT   
  TUMBLE_START(l.ts, INTERVAL '10' SECOND),  
  l.Product, l.Api, l.Version, l.CallerUid, g.region_id, g.app_group, u.gc_level, l.HttpCode, l.ErrorCode,  
  COUNT(*) as qps,  
  AVG(CAST(l.TotalUsedTime AS DOUBLE)),  
  SUM(CASE WHEN l.TotalUsedTime > 1000 THEN 1 ELSE 0 END),  
  SUM(CASE WHEN l.HttpCode >= '200' AND l.HttpCode < '300' THEN 1 ELSE 0 END),  
  SUM(CASE WHEN l.HttpCode >= '500' THEN 1 ELSE 0 END),  
  SUM(CASE WHEN l.HttpCode = '503' THEN 1 ELSE 0 END)  
FROM openapi_log_source l  
LEFT JOIN gateway_cluster_dim FOR SYSTEM_TIME AS OF l.ts AS g ON l.LocalIp = g.local_ip  
LEFT JOIN user_level_dim FOR SYSTEM_TIME AS OF l.ts AS u ON l.CallerUid = u.uid  
GROUP BY   
  TUMBLE(l.ts, INTERVAL '10' SECOND),  
  l.Product, l.Api, l.Version, l.CallerUid, g.region_id, g.app_group, u.gc_level, l.HttpCode, l.ErrorCode;

Job 2: Transform and Aggregate Metrics
Job 2 is deployed in each region to consume the log machine_agg_log, transform data into a time series format, and write the data to a centralized MetricStore in China (Shanghai).

Data Source: Consume a Regional Aggregation Log

CREATE TABLE machine_agg_log_source (  
  window_start TIMESTAMP(3),  
  product STRING,  
  region_id STRING,  
  -- ... Other field definitions are identical to machine_agg_log_sink   
  WATERMARK FOR window_start AS window_start - INTERVAL '5' SECOND  
) WITH (  
  'connector' = 'sls',  
  'project' = '****',  
  'logstore' = 'machine_agg_log',  -- Consume the logstore in the region   
  'endpoint' = 'cn-shanghai-intranet.log.aliyuncs.com'  
);

Sink: Centralized MetricStore Sink

CREATE TABLE metricstore_sink (  
  `__time_nano__` BIGINT,  
  `__name__` STRING,  
  `__labels__` STRING,  
  `__value__` DOUBLE  
) WITH (  
  'connector' = 'sls',  
  'project' = '****',      -- The centralized SLS project   
  'logstore' = 'openapi_metrics',            -- The centralized logstore   
  'endpoint' = 'cn-shanghai-intranet.log.aliyuncs.com' -- The region endpoint   
);

3.Compute and Aggregation Logic
Job 2 performs further aggregation (such as by product), adds the timestamp info, and writes to the centralized project.

Example: Calculate QPS by product and aggregate it

INSERT INTO metricstore_sink  
SELECT   
  UNIX_TIMESTAMP(CAST(TUMBLE_START(window_start, INTERVAL '1' MINUTE) AS STRING)) * 1000000000,  
  'namespace_product_gw_http_req',  
  CONCAT('product=', product, '|region_id=', region_id), -- Retain region info  
  CAST(SUM(qps) AS DOUBLE)  
FROM machine_agg_log_source  
GROUP BY TUMBLE(window_start, INTERVAL '1' MINUTE), product, region_id;

Solution benefits:

Bandwidth savings: Job 1 aggregates massive logs into smaller data (reduced by 99%). Job 2 only transmits these lightweight metrics across regions, which greatly reduces transfer costs.

Isolation: Data processing in each region is independent. A failure in a single region does not affect other regions.

Job Configuration and Optimization
To ensure job stability and data accuracy, we performed special optimization on the checkpoint and state backend in the production environment.

Checkpoint Configuration and Trade-offs
Two checkpointing strategies are provided: one for data consistency, the other for service availability:

Strategy A: Prioritizing data consistency (recommended for general scenarios)

This strategy is applicable to most monitoring scenarios that prioritize data accuracy.

SET 'execution.checkpointing.interval' = '60s';           -- Checkpoint every one minute   
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';      -- Exactly-once semantics   
SET 'execution.checkpointing.timeout' = '10min';

Strategy B: Prioritizing high availability (this example)

Because this example involves highly concurrent data processing and is sensitive to availability, we adopt strategy B to reduce performance jitter from frequent checkpointing, without sacrificing consistency:

SET 'execution.checkpointing.interval' = '180s';          -- Checkpoint at a three-minute interval  
SET 'execution.checkpointing.mode' = 'AT_LEAST_ONCE';     -- Use at-least-once semantics   
SET 'execution.checkpointing.timeout' = '15min';          -- Relax checkpointing timeout   
SET 'execution.checkpointing.max-concurrent-checkpoints' = '1';  
SET 'execution.checkpointing.tolerable-failed-checkpoints' = '10'; -- Tolerate consecutive checkpoint failures to avoid job restart

Strategy comparison:

State Backend
Realtime Compute for Apache Flink provides the enterprise-level GeminiStateBackend. Compared with RocksDB used in Apache Flink, GeminiStateBackend is optimized for large-state jobs under the storage-compute-separation architecture. This example enables GeminiStateBackend and key-value separation to deal with large state and multiple aggregation keys:

SET 'table.exec.state.backend' = 'gemini';                -- Enable GeminiStateBackend  
SET 'state.backend.gemini.kv.separate.mode' = 'GLOBAL_ENABLE'; -- Enable k/v separation

GeminiStateBackend vs. RocksDB:

Production recommendations: For scenarios such as log aggregation with a large state size and extremely high throughput requirements, use GeminiStateBackend and key-value separation. Actual tests show after key-value separation is enabled, the CPU utilization of the job during traffic peaks decreases by 20%, and the checkpoint duration is more stable.

Visualization and Alert
Metric Visualization
A multi-dimensional API monitoring Grafana dashboard is built for deep drill-down analysis, by product or specific error code.

Self-service Query and Alerting
After SLS MetricStore is added as a data source in Grafana, each cloud product team can use Prometheus Query Language (PromQL) syntax to query metrics and configure their own alert rules:

Sample query:

# QPS trend  
sum(namespace_product_gw_http_req) by (product)  

# Error rate (current 1 min vs. 1hr ago)  
(  
  sum(rate(namespace_product_gw_http_5xx[1m])) / sum(rate(namespace_product_gw_http_req[1m]))  
) / (  
  sum(rate(namespace_product_gw_http_5xx[1m] offset 1h)) / sum(rate(namespace_product_gw_http_req[1m] offset 1h))  
) > 2  

# Avg latency   
avg(namespace_product_gw_rt_mean) by (product)

Example alert rule:

- alert: HighErrorRate
  expr: sum(namespace_product_gw_http_5xx) by (product) / sum(namespace_product_gw_http_req) by (product) > 0.01
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "{
   { $labels.product }} error rate is too high"
    description: "Current error rate: {
   {
    $value | printf \"%.2f\" }}%"

Each cloud service team can configure their monitoring dashboard and alert rules in Grafana for autonomous O&M.

Validation in Production
This solution has been stably running in production. Core metrics:

Thanks to the distributed computing capability of Flink and the high throughput storage of SLS, this solution has successfully supported the real-time monitoring of all API calls in Alibaba Cloud Open Platform. It covers more than 60 global regions and more than 300 cloud products, processes more than 200 TB of compressed logs (about 2 PB of raw logs, with a single log being about 4 to 5 KB) per day, and generates over 500,000 time series metrics.

Data Processing Size

Metric Generation Capability

System Stability

Business Benefits
● Rapid fault discovery: The fault discovery time is shortened from minutes to seconds.

● Improved O&M efficiency: More than 300 cloud service teams have achieved self-service monitoring configuration.

During the implementation of the solution, we found the raw log contains a large number of redundant fields and nested structures, whereas metric calculation requires several core fields. To address this, we introduced predicate pushdown at the source for field pruning before data enters Flink, which effectively reduced network transmission and accelerated Flink processing.

Advanced Optimization: Predicate Pushdown
Predicate Pushdown Capability by Connector
Predicate pushdown, a classic database and big data optimization, executes filter conditions at the source. This reduces data volume and compute overhead. Flink's pushdown capability depends on its source connector implementation:

Predicate Pushdown with SPL
In its early versions, the Realtime Compute for Apache Flink connector for SLS pulled all data from an SLS Logstore. But actually, many fields are not needed. SPL enables source-side predicate pushdown by doing filtering and conversion at SLS and sends processed results to Flink.

Benefits:

● SIMD vectorization: SPL's vectorized execution engine uses CPU SIMD instructions (e.g., AVX2/AVX-512) for batch data processing, achieving several times the performance of row-by-row processing.

● Local processing: Data processing is completed on the SLS data node. You do not need to transfer raw data across networks, which avoids network I/O from becoming a bottleneck.

● Columnar storage acceleration: SLS's columnar storage, in combination with column pruning on project, reads only necessary column data. This significantly reduces disk I/O.

● Zero-copy transmission: The processed data directly enters consumption, which reduces the memory copy overhead.

Billing tips:

Non-SPL consumption: billing is based on the transmitted (compressed) data size.

SPL consumption: billing is based on the raw (uncompressed) data size.

For detailed pricing and differences, refer to SLS pricing documentation.

Sample SPL Configuration
This section introduces filtering data with SPL at the source. Consider the traditional approach:

-- Traditional approach: Pull all data and filter with Flink  
SELECT * FROM openapi_log_source  
WHERE Domain != 'popwarmup.aliyuncs.com'  
  AND JSON_VALUE(Ext5, '$.logRegionId') NOT IN ('cn-shanghai', 'cn-beijing')

After SPL is used, filtering and transform are completed on SLS:

-- 1.Row filtering: Exclude invalid data  
*   
| where Domain != 'popwarmup.aliyuncs.com'  

-- 2.Expand nested JSON   
| parse-json -prefix='ext5_' Ext5    
| where ext5_logRegionId not in ('cn-shanghai', 'cn-beijing', 'cn-hangzhou')  
| parse-json -prefix='callerInfo_' ext5_callerInfo    
| parse-json -prefix='headers_' ext5_headers    

-- 3.Extract key-value fields  
| parse-regexp RequestContent, '[;]RegionId=([^;]*)' as request_regionId    

-- 4.Column pruning: Retain necessary fields to reduce output data size  
| project LocalIp, Product, Version, Api, Domain, ErrorCode, HttpCode,   
         TotalUsedTime, AK, RegionId, ClientIp,   
         callerInfo_callerType, callerInfo_callerUid, callerInfo_ownerId,  
         ext5_regionId, ext5_appGroup, ext5_stage, request_regionId

Use SPL
In Flink SQL, reference the pre-configured SPL using the processor parameter:

CREATE TABLE openapi_log_source (  
  `__time__` BIGINT,  
  -- SPL processed fields (JSON object expanded, column pruned)  
  LocalIp STRING,  
  Product STRING,  
  Version STRING,  
  Api STRING,  
  Domain STRING,  
  ErrorCode STRING,  
  HttpCode STRING,  
  TotalUsedTime BIGINT,  
  AK STRING,  
  RegionId STRING,  
  ClientIp STRING,  
  callerInfo_callerType STRING,      -- Get from Ext5.callerInfo  
  callerInfo_callerUid STRING,  
  callerInfo_ownerId STRING,  
  ext5_regionId STRING,              -- Get from Ext5   
  ext5_appGroup STRING,  
  ext5_stage STRING,  
  request_regionId STRING,           -- Get from RequestContent  
  ts AS TO_TIMESTAMP_LTZ(`__time__` * 1000, 3),  
  WATERMARK FOR ts AS ts - INTERVAL '5' SECOND  
) WITH (  
  'connector' = 'sls',  
  'project' = '****',  
  'logstore' = 'pop_rpc_trace_log',  
  'endpoint' = 'cn-shanghai-intranet.log.aliyuncs.com',  
  'processor' = 'openapi-processor'  -- Use SPL for filter pushdown  
);

Optimization Effects
SPL delivers significant improvements in the following areas:

Summary
With the cloud-native solution, we have successfully built a real-time monitoring system for Alibaba Cloud API gateway. Recap:

Flink Highlights

Architectural Design Insights

1.Alleviate data skew: Use layered aggregation: local first, then global by business dimension.
2.Reduce costs with predicate pushdown: Filter at the source (e.g., with SPL) to minimize network transmission and compute.
3.Enterprise-grade state backend: For large states, use GeminiStateBackend with key-value separation for improved I/O and job stability.
The technical solution in this article can be promoted to similar scenarios, such as microservice invocation chain monitoring, Alibaba Cloud CDN log analysis, and Internet of Things (IoT) data aggregation.

References
● Realtime Compute for Apache Flink's SLS connector

● SLS MetricStore

● Send time series data from SLS to Grafana

● SPL syntax

Building Cross-Cloud Observability: One Architecture, Unified Analytics

ObservabilityGuy — Wed, 29 Apr 2026 07:09:49 +0000

This article introduces a unified observability architecture for cross-cloud log analysis and AIOps, designed to streamline multicloud O&M and reduce costs for global enterprises.

1.Customer Requirements
1.1 Unified Analysis of Multicloud Logs
A common form in multicloud scenarios is that edge security and access capabilities outside China are handled by Cloudflare (Web Application Firewall (WAF), Content Delivery Network (CDN), and Access), and Verbose Logs are uniformly stored in Amazon Simple Storage Service (S3) through Logpush for low-cost archiving and compliance retention. Meanwhile, the core business and observability systems of the headquarters often Run on the Alibaba Cloud side. For example, application, gateway, and business logs enter Simple Log Service (SLS), and the alerting, on-call, and ticket systems are also built around the Alibaba Cloud side. The Result is that the "chain of evidence" of the same User Request, the same Attack, or the same publish Change is Distributed across both the Third-party cloud vendor and Alibaba Cloud side. This makes it difficult to complete unified retrieval, association analysis, or closed-loop handling in a single platform.
For the platform engineering team, the core challenge is not the location of log storage, but rather the lack of a unified platform to perform analysis and complete operational tasks.

● Logs are in S3, but troubleshooting, security analytics, and operation Analysis are often scattered across multiple Systems (Cloudflare console, Athena, Glue, Amazon Elastic MapReduce (EMR), CloudWatch, Business Intelligence (BI), and self-built alerting).

● Metrics cannot be standardized: the same Metric (such as 5xx, P99 latency, and WAF block ratio) is calculated separately in different Systems. It is difficult to audit Changes, reuse them, or perform migration.

● The management event response chain is long: it requires "first querying logs -> then manually summarizing -> then sending Notifications -> then dispatching tickets or performing rollback", and the Mean Time To Detect (MTTD) and Mean Time To Resolve (MTTR) are artificially lengthened.

1.2 Reduce Costs and Simplify O&M
If S3 is used as Log Storage, to "use" the Data (query and analysis, visualization, and alerting filter interaction), a combination of additional components is usually required for querying, ETL, metrics, and alerting. The chain becomes longer, configuration and troubleshooting span multiple Systems, and O&M complexity will significantly increase.

If Data is directly connected to CloudWatch: CloudWatch Logs is used for Collection and storage, Logs Insights is used for query and analysis, and Dashboards and Alarms are used for gauge and alerting closed-loops. The overall cost is usually very high.

2.SLS Solutions

Next, the data import, processing, query and analysis, gauge display, and alerting features in this set of SLS Solutions will be broken down and introduced step by step.

2.1 Import Data from S3 to SLS
In the eyes of many people, data import is just the three-step procedure of "read-transmit-write". But when you face:

● Logs that generate thousands of files per minute

● Attack and defense traffic that instantly surges from 1 GB to 10 GB

● Various mixed data formats such as gzip, snappy, JavaScript Object Notation (JSON), and Comma-Separated Values (CSV)

You will find that this is by no means a simple "copy and paste" operation.

Next, the difficulties encountered in the actual import procedure will be clarified first, and then the corresponding implementation methods will be explained:

Challenge 1: The "real-time Search" of massive small files is not simple (full traverse vs. real-time, incremental traverse vs. Integrity)
The ListObjects operation of S3 only Supports traverse in lexicographic order, and does not Support "filtering by Time". When the volume of History files in a bucket or folder is huge, a full scan may take a long Time. However, if only an incremental scan is performed, files may be missed because file names are out of order.

Consequences: New files are not Searched in Time (latency increases), or they are missed in extreme cases (Integrity threat).

Challenge 2: The throughput must be able to keep up with the peak, but cannot rely on "manual parameter tuning" (traffic burst + the "long tail" problem, where processing is slowed down by a few oversized files)

1.In real business, traffic will burst: usually 1 GB/minute, but it may surge to 10 GB/minute during Activities or faults. If the scale-out is slow, the end-to-end latency immediately becomes out of control after the queue accumulates.
2.Even if the concurrent capacity is fully utilized, long tails will still be encountered: "average assign by the number of files" will cause a Job to be dragged down by an oversized file, and the overall latency is determined by the slowest one.
Challenge 3: The data formats are often mixed and unpredictable
The same bucket may often mix JSON, CSV, and text. Even for JSON, it may be "line-by-line JSON, JSON array, or specific service formats (such as CloudTrail)". The compress may be .gz, .snappy, .lz4, or .zstd.
If you attempt to automatically detect the data format, sampling misjudgments and additional read overhead will be Imported, which will slow down the transmission chain instead.
Challenge 4: Data integrity and traceability must be guaranteed (ensuring no data is lost, supporting reprocessing, and enabling problem-file identification)
The import chain naturally has retry and replay: network jitter, Consumption timeout, Job restart, and management events and scans hitting the same object at the same Time may all cause repeated pulls.
More importantly, data loss is often more hidden. Missed events, permission Changes, scan point drifts, and parse abnormalities can cause data gaps during a certain period without being noticed.
Our design solutions for these difficulties are as follows:

● Design point 1: A "dual-mechanism" for file discovery ensures both timeliness and completeness.

SQS Event-driven: S3 events → SQS → data import Job Consumption (suitable for scenarios with irregular file names or low-latency requirements).
Dual-pattern traverse: Incremental catch-up to the latest point + periodic full fallback (to prevent missed discovery).

● Design point 2: Auto Scaling + balanced allocation by data volume to handle traffic peaks and manage long-tail data.

Concurrent Jobs automatically scale out or in based on queues and data volumes. This avoids manual parameter tuning.
Job assignment is upgraded from "by the number of files" to "balanced allocation by data volume". This ensures that a round of concurrent Jobs can be completed at the same time as much as possible.

● Design point 3: Auto compression detection and explicit configuration of data formats (no guessing).

Compression Formats are automatically detected and decompressed based on file suffixes, such as .gz, .snappy, .lz4, and .zstd.
Data formats are explicitly specified by data import Jobs (such as JSON, CSV, single-line, multi-line, CloudTrail, and JSON array). Encoding Settings are also provided (default to UTF-8, and can be specified when necessary). ● Design point 4: Point and Status Management + retry and fencing + file-level tracking to make data backfilling feasible.

On the discovery side, "events + scan fallback" form a compensation closed loop to reduce the probability of missed discovery.
On the pull side, points and processing Status are maintained. Failed files enter the retry or fencing queue. Data backfilling by replaying object keys is Supported.
Deduplication and idempotence control the Impact of duplicates based on object identities (such as key + etag/version + offset) to make duplicates controllable and gaps visible.
2.2 One-stop Data Analytics
Data import is only the first step. A complete observability closed loop also requires data governance, interactive search, visualization, and intelligent alerting. SLS integrates these capabilities into a unified platform. The core principles of each step are described below.

Data transformation: fully managed streaming extract, transform, and load (ETL)
SLS data transformation is based on managed real-time Consumption Jobs and uses Structured Process Language (SPL) syntax to process logs in streams. It is fully managed, supports Auto Scaling, and makes Data visible in seconds. It also Supports line-by-line debugging and code hinting.

SLS uses the SPL engine as the kernel on the log pipeline, which includes advantages such as column-oriented calculation, single instruction multiple data (SIMD) acceleration, and C++ implementation. Based on the distributed architecture of the SPL engine, we have redesigned the Elasticity mechanism. It is not just scaling at the granularity of an instance (such as a Kubernetes pod or service compute unit) in the usual sense, but can quickly scale at the granularity of a DataBlock (MB level).

Scenario capabilities:

● Pre-compliance: IP-to-Geo transform and desensitization are completed outside China. Only compliance fields are retained after cross-border data transfer to meet General Data Protection Regulation (GDPR) and data export requirements.

● Data filtering: Invalid Data is removed to reduce downstream index and storage overheads.

● Structured extraction: Original fields are transformed into analyzable Metrics, and nested JSON is parsed to avoid repeated calculations during queries.

● Field projection: Only Gold fields are delivered, which can reduce cross-border traffic and index costs by 50% to 80%.

● Field enrichment: Field connection (JOIN) is performed on logs (such as order logs) and dimension tables (such as User information Tables) to Add more dimension information to logs for data analytics.

● Data forwarding: Logstore Data can be forwarded and aggregated to destination databases. Data can also be flexibly forwarded based on field Content.

Query and analysis: High-Performance engine and responses in seconds
SLS provides a high-Performance query engine that Supports the index pattern (responses in seconds for tens of billions of Data records) and the scan pattern (lightweight Analysis). Queries are directly applied to indexes without the need to pre-build datasets or wait for purge delays. For ultra-large-scale data analytics scenarios, SLS provides the Dedicated SQL, which includes the enhancement mode and complete accuracy mode.

Query engine and capabilities:

● Nearly a hundred Window Functions: Built-in statistical, aggregation, string, Time, and geospatial functions are provided out-of-the-box.

● Cross-database federated queries: StoreView supports cross-Project and cross-Logstore Data associated queries.

● SQL Exclusive: Provides high-precision Analysis capabilities in large data volume scenarios to avoid sampling errors.

● Scheduled SQL: Supports scheduled execution of SQL queries for Report Generation and Metric pre-computation.

Dashboards: Rich visualization, out-of-the-box
SLS dashboards are Data Visualization Tools provided by Simple Log Service to display query and analysis Results in a graphical interface. A dashboard usually contains multiple statistical charts to summarize and render key performance metrics, important Data, and Analysis Results.

Visualization capabilities:

● Rich chart Types: Multiple statistical charts such as Tables, line charts, column charts, pie charts, and maps are supported. The Pro Version supports the overlaid display of multiple query Results.

● Interaction and drill down: Supports global Time filtering, variable filter interaction, and chart drill down to track from the overall situation to details layer by layer.

● Subscribe and Share: Supports periodically rendering dashboards into Images and sending them by Email or to DingTalk groups. Supports embedding the console into third-party Systems.

● Third-party Integration: Can be integrated with visualization tools such as DataV, Grafana, and Tableau, and supports bidirectional import and export of Grafana dashboards.

Alerting: A one-stop artificial intelligence for IT operations platform
SLS alerting is a one-stop artificial intelligence for IT operations platform for alerting and monitoring systems, denoising, transaction management, and Notification dispatch. It consists of subsystems such as the alerting and monitoring system, alert management system, and notification management system. After logs or metrics are ingested, you can create monitoring jobs, notification channels, and alert policies within minutes.

Feature advantages:

● Low cost and fully managed: Provided as Software as a Service (SaaS). Except for text messages and voice calls, no additional fees are charged for alerting and monitoring systems, transaction management, or other features.

● Denoising and dispatch: Supports grouping, removing duplicates, suppression, and upgrading to avoid alert storms. Supports automatic dispatch to different teams based on rules.

● Rich notification channels: Natively integrates DingTalk, WeCom, Lark, Slack, text messages, voice calls, and Webhooks.

2.3 O&M Simplification (Using Integration to Replace Multiple Product Portfolios)
2.3.1 THIRD-PARTY CLOUD VENDOR multiple product portfolios: Which components are usually required to achieve the same closed loop

Having multiple components is not necessarily bad, but when your requirement is "unified standards, minute-level closed loop, and controllable low cost," multiple components mean:

● Longer pipeline: Data needs to be moved more times (extract, transform, and load (ETL), saving to intermediate tables, and refreshing datasets).

● Larger failure surface: Jitter in any step will Impact the end-to-end timeliness.

● More fragmented billing: Costs for storage, scans, ETL, alerting, visualization, and Networks are all increasing.

2.3.2 SLS integration vs THIRD-PARTY CLOUD VENDOR multiple product portfolios
In SLS, you can create a reusable engineering template that combines "import + processing + index + query + dashboard + alerting/transaction," use the template to deliver the first version, and use policies to iterate on costs and results.

3.Case Study of Log Analysis Architecture Upgrades for Globalized Enterprises
Background and Solutions
A large globalized enterprise whose business covers multiple areas such as Europe, Asia-Pacific, and North America achieves global access acceleration and Web application protection through mainstream Alibaba Cloud CDN and security services. To meet Data compliance and audit requirements outside China, the enterprise continuously archives its security and access logs to public cloud Object Storage Service for long-term retention and subsequent Analysis through the native log push capability (Logpush) of the platform.

Currently, the enterprise uses a combination of multiple components on THIRD-PARTY CLOUD VENDOR to achieve the Analysis and monitoring of logs outside China, and encounters the following problems:

● Scattered Data: S3 is distributed across multiple Regions such as Frankfurt and Tokyo, and data silos are difficult to uniformly manage and analyze.

● High query and analysis costs: Athena bills based on scan volume. CloudWatch Logs Insights has limited query capabilities and requires separate queries across regions. The costs of daily retrievals and alerting queries increase linearly with frequency.

In addition, extract, transform, and load (ETL) dependencies on Glue or Lambda require self-maintenance. QuickSight visualization requires additional authorization and has synchronization latency. CloudWatch Alarms configurations are scattered and lack unified denoising capabilities. The multiple product portfolio causes issues such as high O&M complexity and uncontrollable costs.

You can build a unified observability analysis platform based on SLS to achieve the following goals:

● Unified data transformation: You can use Structured Process Language (SPL) to complete data governance outside China (such as field clipping, IP address desensitization, and Geo enrichment). This reduces the costs of cross-border transfer.

● Unified query and analysis: You can aggregate gold data in the central Logstore in China to provide second-level interactive search for hundreds of millions of data records.

● Unified visualization: A one-stop dashboard is provided, and no additional business intelligence (BI) tools are required.

● Unified alerting closed loop: Intelligent alerting based on SLS query and analysis is provided. It supports denoising, dispatching, and multi-channel notifications.

3.1 Data Flow
Data is pushed from Cloudflare Logpush to various Amazon Web Services (THIRD-PARTY CLOUD VENDOR) S3 regions outside China for archiving. SLS imports the data into Logstores in the same region through event-driven mechanisms or scheduled scans. After the data is transformed by SPL, it is aggregated into the central Logstore in China to support unified query and analysis, dashboards, and alerting.

3.1.1 Sample SPL data transformation
Sample raw log (Cloudflare Web Application Firewall (WAF) log)

The sample Cloudflare WAF raw log contains sensitive and security fields such as ClientIP, SecurityAction, and SecuritySources, and covers three security action scenarios: block, allow, and challenge. You can directly use these logs to test SPL data transformation statements.

{  
  "EdgeStartTimestamp": "2024-12-25T10:30:00Z",  
  "RayID": "abc123def456",  
  "ClientIP": "203.0.113.50",  
  "OriginIP": "10.0.0.100",  
  "ClientRequestURI": "/api/v1/users?id=123",  
  "ClientRequestMethod": "POST",  
  "ClientRequestReferer": null,  
  "SecurityAction": "block",  
  "SecurityRuleID": "rule_001",  
  "SecuritySources": "[{\"source\":\"waf\",\"action\":\"block\"}]",  
  "OriginResponseStatus": 200,  
  "OriginResponseTime": 150,  
  "ResponseHeaders": "{\"x-cache\":\"MISS\"}"  
}

The following SPL script completes data governance outside China: time standardization, IP address to Geo geographic information conversion, IP address desensitization to anonymous fingerprints, security metadata parsing, and threat labeling. Finally, sensitive fields such as ClientIP and OriginIP are removed by using project-away, and only gold fields are retained for cross-border transfer.

-- Core tracking and time standardization  
*   
| extend __time__ = cast(to_unixtime(date_parse(EdgeStartTimestamp, '%Y-%m-%dT%H:%i:%SZ')) as bigint)  
| extend RequestId = RayID  
| extend RequestPath = url_extract_path(ClientRequestURI)  

-- IP -> Geo (completed outside China)  
| extend  
    GeoCountry = ip_to_country(ClientIP),  
    GeoRegion  = ip_to_province(ClientIP),  
    GeoCity    = ip_to_city(ClientIP)  

-- IP address desensitization: Retain anonymous fingerprints (optional) and do not carry the raw IP address for cross-border transfer  
| extend ClientFingerprint = to_base64(sha256(to_utf8(ClientIP)))  

-- Security metadata parsing and labeling  
| expand-values -keep SecuritySources  
| parse-json -prefix='Security' SecuritySources  
| extend IsHighRisk = if(ClientRequestMethod = 'POST' and (ClientRequestReferer is null or SecurityAction = 'block'), 1, 0)  

-- Final denoising and field projection  
| project-away ClientIP, OriginIP, ResponseHeaders, RayID

Sample Data after data transformation

The data after data transformation has completed Geo enrichment, IP masking, and threat labeling. Sensitive fields have been removed, and the data can be directly used for downstream query and analysis and alerting:

{  
    "RequestPath": "/api/v1/users",  
    "__time__": "1735122600",  
    "RequestId": "abc123def456",  
    "ClientFingerprint": "O1zTaFfLyH1ZqEHS03UiLSNMzwMX+4ZW7OsIVsDGgEg=",  
    "OriginResponseTime": "150",  
    "GeoCity": "Richardson",  
    "ClientRequestURI": "/api/v1/users?id=123",  
    "IsHighRisk": "1",  
    "EdgeStartTimestamp": "2024-12-25T10:30:00Z",  
    "SecurityAction": "block",  
    "SecurityRuleID": "rule_001",  
    "Securityaction": "block",  
    "GeoCountry": "United State",  
    "GeoRegion": "Texas",  
    "OriginResponseStatus": "200",  
    "Securitysource": "waf",  
    "ClientRequestMethod": "POST"  
}

3.1.2 query and analysis samples
Sample 1: Web Application Firewall (WAF) rule hit Statistics - This sample aggregates the hit Count, high-threat proportion, and unique attacker count by rule.

* | SELECT   
  SecurityRuleID,  
  count(*) AS TotalHits,  
  count_if(IsHighRisk = 1) AS HighRiskHits,  
  approx_distinct(ClientFingerprint) AS UniqueClients  
FROM log  
WHERE SecurityRuleID IS NOT NULL AND SecurityRuleID <> ''  
GROUP BY SecurityRuleID   
ORDER BY TotalHits DESC

Sample 2: Top 10 Attack source regions - This sample aggregates the block Count and unique attacker count by country or city.

* | SELECT   
  GeoCountry,  
  GeoCity,  
  count(*) AS AttackCount,  
  approx_distinct(ClientFingerprint) AS UniqueAttackers  
FROM log  
WHERE SecurityAction = 'block'  
GROUP BY GeoCountry, GeoCity  
ORDER BY AttackCount DESC  
LIMIT 10

Sample 3: Origin 5xx fault Trend - This sample aggregates the fault Count, Error Rate, and total Request count by minute.

* | SELECT   
  time_series(__time__, '1m', '%Y-%m-%d %H:%i:%s', '0') AS TimeBucket,  
  count_if(OriginResponseStatus >= 500) AS Origin5xxCount,  
  count_if(OriginResponseStatus >= 500) * 100.0 / count(*) AS Origin5xxRate,  
  count(*) AS TotalRequests  
FROM log  
GROUP BY TimeBucket  
ORDER BY TimeBucket

Sample 4: Request latency quantile Analysis - This sample aggregates P50/P90/P99 latency by path to locate slow APIs.

* | SELECT   
  RequestPath,  
  count(*) AS RequestCount,  
  approx_percentile(OriginResponseTime, 0.50) AS LatencyP50,  
  approx_percentile(OriginResponseTime, 0.90) AS LatencyP90,  
  approx_percentile(OriginResponseTime, 0.99) AS LatencyP99  
FROM log  
WHERE OriginResponseTime IS NOT NULL  
GROUP BY RequestPath  
HAVING count(*) > 100  
ORDER BY LatencyP99 DESC  
LIMIT 20

3.1.3 Alert rule samples
Alert 1: Sudden increase in origin 5xx faults - This alert is triggered when the Error Rate exceeds 5% to rapidly discover origin abnormalities.

* | SELECT  
    count_if(OriginResponseStatus >= 500) * 100.0 / count(*) AS Origin5xxRate  
  FROM log  
  HAVING Origin5xxRate > 5

Alert 2: Sudden increase in high-threat Requests - This alert is triggered when the Count exceeds 100 or the proportion exceeds 10% to detect potential Attacks.

* | SELECT  
    count_if(IsHighRisk = 1) AS HighRiskCount,  
    count_if(IsHighRisk = 1) * 100.0 / count(*) AS HighRiskRate  
  FROM log  
  HAVING HighRiskCount > 100 OR HighRiskRate > 10

Alert 3: Sudden increase in WAF blocks - This alert is triggered when the block Count exceeds 1000 or the unique attacker count exceeds 50 to assess the attack posture.

* | SELECT  
    count_if(SecurityAction = 'block') AS BlockCount,  
    approx_distinct(ClientFingerprint) AS UniqueAttackers  
  FROM log  
  HAVING BlockCount > 1000 OR UniqueAttackers > 50

4.Summary and Outlook
During the data migration procedure, the network quality and fees of cross-cloud and Cross-border Transfer cannot be ignored. Therefore, we have implemented the capability to reduce the overhead of cross-cloud and Cross-border Transfer by using CloudFront for users to choose.

References
● Import data from Amazon S3 to Simple Log Service

● THIRD-PARTY CLOUD VENDOR Glue Pricing

● Simple Log Service Pricing

One Command Equips Your OpenClaw with an X-ray Machine - Alibaba Cloud Observability Makes Farming Lobsters Cheaper and Safer

ObservabilityGuy — Tue, 28 Apr 2026 02:08:48 +0000

One-command observability integration makes OpenClaw AI agent operations transparent via Alibaba Cloud monitoring plugins.

❓Have you experienced this?

OpenClaw🦞(an open-source AI agent framework) is becoming a "digital employee" for more enterprises. It processes emails, writes code, manages files, and executes commands. It does almost anything. Many teams have deployed dozens or hundreds of OpenClaw instances. They formed a sizable "digital lobster farm".

However, a problem arises.

Lobster farmers can at least watch their pond. What about your OpenClaw? Do you know how many tokens it consumed today? Do you know which model is silently draining your budget? Do you know if a "lobster" was lured into reading /etc/passwd at 3:00 AM?

The answer for most is: I don't know. 😶

You carefully deployed OpenClaw. However, when these issues arise, you find yourself without the right tools to pinpoint the problem.

This article discusses using one command to equip your OpenClaw with an X-ray machine. This makes every LLM invocation, tool execution, and token consumption visible.

1.What Is Your Lobster Doing? Three “Blind Spots” Are Affecting Your Confidence
📚 Before we start, let's discuss three "blind spots". If you use OpenClaw, at least one has likely troubled you.

Blind spot 1: The inference process is a maze and debugging relies on guessing
The complete path OpenClaw takes to process a user message is more complex than you think. A simple question may travel the following journey:

User input → System prompt assembly → Model inference round 1 → Determine need for tool calling → Tool calling (such as search or code execution) → Return tool result → Model inference round 2 → Call another tool → Generate final response

If any step fails, the final output may deviate from expectations. Without tracing analysis, you face an "input-output" black box. You can only guess where the problem lies. Is the prompt poor? Is it model hallucination? Did the tool return incorrect data?

Tuning prompts relies on inspiration. Troubleshooting relies on luck. This is not science. It is mysticism. 🎲

Blind spot 2: Token bills are like blind boxes and cause pain at month-end
LLMs charge by token. Everyone knows this. However, as an agent, OpenClaw has a token consumption pattern different from directly invoking an API. It has a context snowball effect.

In every conversation round, the agent stuffs previous conversation history, system prompts, and tool calling results into the context. The first round might use 2000 tokens. By the fifth round, it might expand to 20,000. If a tool returns a large block of HTML or JSON, the situation worsens.

Worse, you do not know the source of the cost. Is a model too expensive? Is an agent prompt too wordy? Was the context not clipped in time? Without fine-grained consumption data, you cannot perform optimization. 💸

Blind spot 3: System status is like Schrödinger's cat
OpenClaw involves message queues, webhook processing, and session management during operation. When a user asks why it is not responding, the problem could lie in any layer. Did model inference timeout? Did tool calling stall? Are message queues stacked? Did the gateway fail?

Without real-time metric monitoring, you only discover issues after user complaints. By then, a group of users may be affected. ⏰

2.The Antidote Is Here: openclaw-cms-plugin + diagnostics-otel, Traces and Metrics Working Together
🛠️ To address these three "blind spots", our solution involves two plugins working together. They solve problems at different layers:

Both rely on the OpenTelemetry standard protocol. Data is uniformly reported to Cloud Monitor 2.0 of Alibaba Cloud. View and analyze data on the same platform.

The openclaw-cms-plugin is the focus of this topic. It is a trace reporting plugin designed for OpenClaw. It follows OpenTelemetry GenAI semantics and generates structured traces for every OpenClaw run.

Specifically, it records the following types of spans:

These spans have a parent-child relationship. Together, they form a complete trace. You can see a trace view similar to this in the Cloud Monitor 2.0 console:

You can see at a glance how many times the LLM was invoked and how many tokens were used. You can also see which tools were invoked, which step took the longest, and if any errors occurred.

It is that simple to go from "guessing" to "seeing". 👁

diagnostics-otel is a built-in extension of OpenClaw. It outputs runtime metrics data, including token consumption rate, invocation QPS, response duration distribution, queue depth, and session status. The installation script automatically finds and enables it. You do not need to do anything else.

Wait, does diagnostics-otel not also report traces? Why is openclaw-cms-plugin needed?
Good question. The diagnostics-otel supports trace reporting. However, if you look closely at the generated trace, you will find a fundamental problem: All spans are independent and have no parent-child relationship.

The diagnostics-otel uses an event-driven architecture to generate spans. Each event creates a span independently with a different trace ID. It generates the following five types of spans:

● openclaw.model.usage: model invocation (records token usage)

● openclaw.webhook.processed/openclaw.webhook.error: webhook processing

● openclaw.message.processed: message processing (records processing results and duration)

● openclaw.session.stuck: session stuck alerting

There is no trace context propagation between these spans. Simply put, they are just independent data points. The only way to associate them is using business fields such as sessionKey.

Webhook  [openclaw.webhook.processed]  traceId: abc123  
Message  [openclaw.message.processed]  traceId: def456  ❌ Different trace IDs  
Model    [openclaw.model.usage]        traceId: ghi789  ❌ Different trace IDs

However, openclaw-cms-plugin is designed for complete tracing. All spans share the same trace ID. They are linked into a call tree via an explicit parent-child relationship. You can see the full picture of a request:

enter_openclaw_system              traceId: aaa111  
  └── invoke_agent main            traceId: aaa111  ✅ Same trace ID  
        ├── chat qwen3-235b        traceId: aaa111  ✅ Same trace ID  
        ├── execute_tool search    traceId: aaa111  ✅ Same trace ID  
        └── execute_tool exec      traceId: aaa111  ✅ Same trace ID

In addition to trace integrity, there is a fundamental difference in data richness between the two:

Simply put: The trace from diagnostics-otel is a set of independent "record cards", while the trace from openclaw-cms-plugin is a complete "invocation map". The former only tells you "what happened," while the latter tells you "every step." Use them together. One handles system metrics, and the other handles business traces. They complement each other perfectly. 🤝

3.Setup in One Minute: One-Command Integration Tutorial
🚀 Enough theory. Let's get started. The entire integration process takes less than a minute.

3.1 Get the install command
Log on to the Cloud Monitor 2.0 console. Go to your application monitoring workspace. Choose Integration Center > AI application observability. Click OpenClaw.

In the sidebar, enter the application name and click Click to obtain to generate the integration command immediately. Click the icon in the upper-right corner to copy it with one click.

3.2 Start installation with one command
Open the terminal on the machine where OpenClaw runs. Paste the command you copied and press Enter:

curl -fsSL https://arms-apm-cn-hangzhou-pre.oss-cn-hangzhou.aliyuncs.com/openclaw-cms-plugin/install.sh | bash -s -- \  
  --endpoint "https://Your ARMS-OTLP address" \  
  --x-arms-license-key "Your license key" \  
  --x-arms-project "Your project" \  
  --x-cms-workspace "Your workspace" \  
  --serviceName "Your service name"

Then, sit back and watch it run. ☕

The installation script automatically does the following:

[INFO]  Checking prerequisites...  
[OK]    Node.js v24.14.0  
[OK]    npm 11.9.0  
[OK]    OpenClaw CLI found  
[INFO]  Downloading plugin...  
[OK]    Downloaded  
[INFO]  Extracting...  
[OK]    Extracted  
[INFO]  Installing npm dependencies...  
[OK]    Dependencies installed  
[INFO]  Locating diagnostics-otel extension...  
[OK]    Found diagnostics-otel at: /home/.../extensions/diagnostics-otel  
[OK]    diagnostics-otel dependencies already present  
[INFO]  Updating config...  
[OK]    Config updated  
[INFO]  Restarting OpenClaw gateway...  
[OK]    Gateway restarted  

════════════════════════════════════════════════════  
  ✅ openclaw-cms-plugin installed successfully!  
════════════════════════════════════════════════════

What does it do?
✅ Checks the environment (Node.js, npm, OpenClaw CLI).
✅ Downloads and decompresses openclaw-cms-plugin to the OpenClaw extension folder.
✅ Installs runtime dependencies for the plugin.
✅ Automatically locates the diagnostics-otel extension. If dependencies are missing, it installs them automatically.
✅ Updates the openclaw.json configuration (configurations for both plugins are written at once).
✅ Restarts the gateway to apply the configuration.
You do not need to manually edit any configuration files. The installation script intelligently handles various edge cases. It merges updates into existing configurations instead of overwriting them. It also searches for multiple possible installation locations for diagnostics-otel based on priority.

3.3 Verify installation
After installation, chat with your OpenClaw. Wait a minute or two. Open the Cloud Monitor 2.0 console. Go to AI application observability in the sidebar on the right. Your OpenClaw application appears. Congratulations. Your lobster is no longer a black box. 🎉

3.4 Want to uninstall? It is even simpler
If you want to stop using it (though I doubt it), one command does it:

curl -fsSL https://arms-apm-cn-hangzhou-pre.oss-cn-hangzhou.aliyuncs.com/openclaw-cms-plugin/uninstall.sh | bash

The uninstall script automatically cleans up the plugin folder and all related configurations in openclaw.json. It also disables the diagnostics-otel configuration. If you only want to uninstall the trace plugin but keep metrics, add the --keep-metrics parameter.

Clean and quick. No side effects. 🧹

4.The Highlight: What Can You See After Installation?
📈 Integration is just the beginning. The truly exciting part is what you see and solve after integration.
4.1 Complete trace: Finally understand its "thought process"
This is the core value of openclaw-cms-plugin. Cloud Monitor 2.0 displays a structured trace for every user request:

enter_openclaw_system (Request entry: sender and source)
　└── invoke_agent main (Agent execution procedure)
　　　├── chat qwen3-235b  (LLM invoke: model inference + token usage details) 
　　　├── execute_tool search (Tool calling: search)
　　　└── execute_tool exec (Tool calling: code execution)

In a conversation round, the plugin records agent-level LLM invokes and each independent tool calling. If the agent runs a tool loop internally (such as "invoke tool → get result → invoke next tool"), each tool calling is recorded independently as a tool span. This includes input parameters, return values, and execution status. You can clearly see the complete toolchain execution procedure.

💡 In the current version, LLM invokes in a conversation round aggregate into one LLM span. It records the final total token usage and input/output content for that round. Future versions will refine this. They will support generating a separate span for each independent LLM inference. Then, even intermediate inference steps in multi-round tool loops will be fully visible.

Each span is annotated with rich properties:

● Duration—see which step is slowest at a glance

● Model information—which model and provider were used

● Token usage—input_tokens, output_tokens, cache_read_tokens, and total_tokens, broken down item by item

● Tool parameters and return values—what tool was invoked, what parameters were passed, and what results were returned

● Error message—displayed in red if an error occurs

What does this mean?

Previously, if a user said the "answer is wrong," you had to guess by checking chat records. Now, check the traces. You see the search tool returned an empty result. The model "creatively" made up a paragraph based on that empty result. Problem localization drops from "two hours" to "two minutes". ⚡

4.2 Token usage breakdown—know exactly where every penny goes
Each LLM span in trace carries complete token usage properties:

Use gen_ai.request.model and gen_ai.provider.name. You can know exactly: which model consumed how many tokens at which step.

Consider a real scenario. You find five LLM invocations in a conversation trace. The input_tokens for the third invocation reach 12,000. Click it. You see the tool returned a full page of HTML, all stuffed into the context. You found the "token-swallowing blackhole." Optimization now has a direction.

Token usage transforms from a "messy account" to a "detailed ledger". 💰

4.3 System running metrics—pulse visible in real-time
Metrics data exported by the diagnostics-otel plugin can build running metric gauges on Cloud Monitor 2.0. This allows real-time monitoring:

● Token usage rate and fee trends — broken down by model and time dimension

● Invoke QPS and response duration — is system throughput normal

● MSMQ depth and wait time — is there a backlog

● Session stall count — Are any lobsters "playing dead"?

● Context size trend — Is the context expanding uncontrollably?

Paired with the alerting feature of Ccloud Monitor 2.0, these metrics enable automatic alerts for a 50% day-over-day surge in daily token consumption, automatic alerts when queue depth exceeds a threshold, and automatic alerts for session stalls. You know immediately when a problem occurs, rather than waiting for user complaints. 🔔

4.4 GenAI semantic conventions — Professional standards, not ad hoc solutions
Note that the trace data reported by openclaw-cms-plugin strictly follows the OpenTelemetry GenAI semantic conventions. These are not field names we defined arbitrarily, but international standards.

This means:

Standardized data structures — Property names such as gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.tool.name match industry standards. This simplifies integration with other tools.
Normalized message formats — gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions are formatted according to standard JSON schema. This supports multiple message types, such as TextPart, ReasoningPart, ToolCallRequestPart, and ToolCallResponsePart.
Future extensibility — As GenAI semantic conventions evolve, the plugin allows smooth upgrades.
4.5 Beyond standards — The "extra helpings" of Alibaba Cloud GenAI conventions
While compatible with OTel open-source standards, openclaw-cms-plugin also implements extension capabilities from the Alibaba Cloud GenAI semantic conventions. Compared to the community Standard Edition, you receive some "extra helpings":

ENTRY span — A clear "entry point" for the trace

The OTel community specification defines only span types such as LLM (inference), tool (tool calling), and agent. It lacks an "entry point" concept. The Alibaba Cloud specification extends the ENTRY span type to specifically identify the call entry point of an AI application. In openclaw-cms-plugin, this is the enter_openclaw_system span. It records "who initiated the request" (gen_ai.user.id) and the "current session ID" (gen_ai.session.id). This lets you view the trace and perform analysis and tracking by user and session dimensions.

🔗 Session-level association —gen_ai.session.id

The OTel standard provides gen_ai.conversation.id. However, for agent applications, "session" is more appropriate than "conversation". The Alibaba Cloud specification introduces gen_ai.session.id, which spans ENTRY, AGENT, and LLM spans. This lets you search directly by session ID in Cloud Monitor 2.0, retrieve all traces under that session at once, and quickly restore the full session content.

📊 gen_ai.span.kind — An AI-specific span categorization system

The SpanKind in the OpenTelemetry standard includes only generic types such as CLIENT, INTERNAL, and SERVER. For an AI application trace, SpanKind alone cannot distinguish between an LLM inference and a tool calling. Alibaba Cloud introduces the gen_ai.span.kind property to define a GenAI-specific classification system: LLM, TOOL, AGENT, ENTRY, TASK, STEP (ReAct round), CHAIN, RETRIEVER, and RERANKER. Cloud Monitor 2.0 uses this categorization to automatically detect the AI application structure and render a dedicated AI trace view. LLM calls appear in orange, tool calling in pink, and agents in green. This lets you see the "role distribution" of the entire trace at a glance.

💡 These extensions do not disrupt standard compatibility. The data reported by openclaw-cms-plugin displays basic information normally on any backend that supports OpenTelemetry. However, Cloud Monitor 2.0 unlocks the complete AI application observability experience.

This standardized approach benefits future data analytics and platform evolution.

5.From Black Box to Transparent: How Observability Changes Your Lobster Farming
📈 Installing an X-ray machine fundamentally changes your "lobster farming" method:

This is not merely an improvement. It is a leap from "blind farming" to "precision farming."

A farmer upgrades from "checking water color visually" to using "water quality sensors, cameras, and automatic feeding systems." You manage the same lobsters, but your control level changes completely. 🦞📊

One more thing: Security audit
Beyond performance tuning and cost control, enterprise AI agent deployment involves an unavoidable topic: security compliance and behavior audit. Agents can execute commands, read and write files, and initiate network requests. Without behavior audit capabilities, you cannot know if an agent secretly read an SSH key at 3:00 a.m.

Our observability team covers this capability with another solution: the Alibaba Cloud Simple Log Service (SLS) OpenClaw one-click solution. It collects OpenClaw session audit logs and application operational logs. It provides out-of-the-box security audit dashboards, including high-risk command detection, prompt injection detection, and sensitive data leakage analysis. This makes every agent operation traceable.

If you are interested in security audits, read this article: https://www.alibabacloud.com/help/sls/enable-managed-openclaw-with-sls (SLS one-click integration and audit solution makes OpenClaw controlled operation possible).

Cloud Monitor 2.0 manages performance and cost, and SLS manages security and compliance. Together, they form a complete control system for the "lobster farm." 🔐

6.FAQs
💡 Here are answers to common questions about the process:

Q: Does the integration impact OpenClaw performance?

A: The impact is minimal. The openclaw-cms-plugin uses the OpenTelemetry batch export mechanism. Span data is buffered in memory and reported in batches periodically. This does not block the normal processing flow of the agent.

Q: Can I install only traces without metrics?

A: Yes. Add the --disable-metrics parameter during installation to skip the diagnostics-otel configuration.

Q: Do traces from diagnostics-otel conflict with traces from openclaw-cms-plugin?

A: The installation script sets diagnostics.otel.traces to false by default. The openclaw-cms-plugin handles trace reporting. They work independently without duplication.

Q: I have configured diagnostics-otel. Will the installation overwrite my configuration?

A: No. The traces, logs, sample rate, and other configurations remain unchanged. It adds necessary fields such as endpoints and headers.

Q: Which OpenClaw versions are supported?

A: The version must be 26.2.19 or later (earlier versions exclude the diagnostics-otel plugin). The openclaw-cms-plugin works using the standard OpenClaw Hook mechanism. It does not depend on internal APIs of specific versions.

Q: Why is the token consumption always 0?

A: OpenClaw introduced a bug in V2026.3.8. This causes incorrect token consumption collection. We are urging the community to expedite the fix. Relevant issue link: https://github.com/openclaw/openclaw/issues/46616

7.Summary
📋 Back to the first question: Do you know what your lobster is doing underwater?

If the answer is "I don't know", it is time to install an X-ray machine.

The openclaw-cms-plugin + diagnostics-otel, and one command: ten minutes to integrate, bringing three core capabilities to your OpenClaw:

✅Tracing analysis— End-to-end visualization of every LLM invocation, tool execution, and token flow.

✅Real-time metrics— Monitor system pulse in real time, including token consumption rate, invocation QPS, queue depth, and session status.

✅GenAI semantic standards— Standardized data structures. They lay the foundation for cost analysis, performance optimization, and exception detection.

Stop letting your lobster "freestyle" in a black box. Install an X-ray machine. Make every step visible, traceable, and optimizable.

After all, a visible lobster is a good lobster. 🦞✨

❓Interaction time!

What is the most troublesome "black box problem" you encountered while using OpenClaw?
How do you troubleshoot OpenClaw issues now? Do you have any hacks to share?
What data do you want to see most after enabling observability?
Share your "lobster farming" insights in the comments. Bring your questions. We are here! 🦞🎉

Accepted by Top Conferences! Multiple Alibaba Cloud Achievements Improve O&M Intelligence Accuracy and Efficiency

ObservabilityGuy — Fri, 24 Apr 2026 06:32:07 +0000

This article introduces three top-conference-accepted research achievements by Alibaba Cloud that solve core AIOps challenges in data augmentation, se...

As the core direction of enterprise digital transformation and artificial intelligence for IT operations (AIOps), operation intelligence is becoming a key enabler for improving business stability and reducing O&M costs in the AI-native era. Its technical development and engineering implementation always revolve around core aspects such as data processing, semantic understanding, and exception detection.

The Alibaba Cloud Observability team continues to work deeply in this field. Recently, a series of research achievements in the operation intelligence realm jointly published with universities such as Fudan University, Tsinghua University, and Tongji University have been consecutively accepted by top international academic conferences International Conference on Learning Representations (ICLR) 2026, Transactions on Software Engineering (TSE) 2026, and International Symposium on Software Testing and Analysis (ISSTA) 2025. These achievements systematically overcome core technical challenges in realms such as metric data augmentation, large-scale semantic parsing, and cross-system exception detection. They build a complete operation intelligence technical system from data infrastructure to semantic understanding, and then to industrial-level deployment. This further promotes the engineering implementation of large language model (LLM) in scenarios such as automatic inspection by AI agents, assisted root cause analysis, and automatic fault recovery. This lays a solid technical foundation for large-scale applications.

Three Major Challenges in the Engineering Implementation of AIOps
Challenge 1: Semantics Gap
Traditional tools process O&M data essentially by performing "format matching". Log resolvers categorize similar strings into one class. Timing analysis applies common methods in the image realm. Exception detection only looks at a single metric. These methods do not understand the essential difference between "timeout after 30s" and "timeout after 0.01s" in the O&M context. They do not understand the statistical semantics such as the trend, epoch, or stationarity of metrics. They also do not know the deep association among logs, metrics, or traces. The lack of semantics directly leads to persistently high missed detections and false positives.

Challenge 2: Generalization Bottleneck
Real O&M systems are never static. Microservices frequently release new versions, and log templates continuously evolve. After new operational systems are published, all history annotations become invalid. The data distribution drifts over time, and the model that was well-trained yesterday may fail today. More critically, the annotation cost of industry-level systems is extremely high. For each new system annotated, it often requires months of human effort. Existing methods perform excellently in a stable lab environment. However, they struggle to adapt to a dynamically evolving production environment.

Challenge 3: Industrial Availability
The academic community pursues accuracy. The industrial community requires both accuracy and efficiency. Log streaming of 100,000 logs per second, abnormal response requirements within 100 ms, and limited memory and computing power budgets are hard constraints. These hard constraints keep many "good methods in papers" confined to the lab. They cannot be truly implemented.

Systematic Breakthroughs of Alibaba Cloud Observability
① AutoDA-Timeseries: Break through the limitations of timing modeling, enabling AI to predict faults with less data
Without a good augmentation policy, the true potential of metrics cannot be tapped. For a long time, metric data augmentation has been limited by paradigm migration in the image domain. Timing features are ignored. Augmentation policies cannot be adaptive. Existing Automated Data Augmentation (AutoDA) frames blindly apply image transformations. This destroys autocorrelation and time dependencies. This critically restricts the performance of downstream tasks such as categorization, prediction, and exception detection.

The paper "AutoDA-Timeseries: Automated Data Augmentation for Time Series" (Tsinghua University & Alibaba Cloud) accepted by ICLR 2026 proposes the first general automated data augmentation frame for metrics. It fetches 24-dimensional timing statistical features and integrates them into a stacking augmentation layer. Through Gumbel-Softmax differentiable sampling, it adaptively optimizes the augmentation probability and intensity in a single-stage end-to-end manner. It covers five major jobs such as categorization, long- and short-term prediction, regression, and exception detection. The categorization accuracy reaches 0.730 (+6.7%) on Temporal Convolutional Network (TCN) and 0.721 (+5.2%) on ROCKET. It comprehensively surpasses 7 state-of-the-art (SOTA) baselines. This provides the first generalized and automated solutions for metric data augmentation.

Paper address: https://openreview.net/forum?id=vTLmHAkoIW

② A SemanticLog: Balancing high accuracy and high throughput, the peak throughput of semantic log parsing reaches 1.28 million logs per second
Without good semantic understanding, the true meaning behind log parameters cannot be read. Log parsing technology has remained at the syntax layer for a long time. That is, it uniformly replaces dynamic parameters with the wildcard character (*). This loses semantic information carried by parameters, such as object identifier (ID), status code, and UNIX timestamp. This critically restricts the accuracy of AIOps downstream tasks such as exception detection and root cause analysis. Existing LLM resolvers mostly depend on the online APIs of ChatGPT. They face three major challenges: privacy leakage, unstable latency, and uncontrollable versions. They are difficult to implement in a production environment.

The paper "SemanticLog: Towards Effective and Efficient Large-Scale Semantic Log Parsing" (Fudan University & Alibaba Cloud & Tongji University), accepted by TSE 2026, proposes the first semantic log resolver based on an open-source LLM. The semantic log resolver consists of three core modules that work together. LogLLM removes causal masks and reconstructs log parsing from text generation to a token categorization job to fully utilize bidirectional context. The SemPerception module uses multi-head cross-attention to aggregate subword features and achieves 16 classes of fine-granularity semantic categorization (which is extended by 60% compared to the VALB 10-class system, and 96% of parameters in enterprise logs can be accurately categorized). The EffiParsing prefix tree caches parsed templates to significantly reduce repetitive inference overhead.

A comprehensive evaluation based on LLaMA2-7B on the LogHub-2.0 benchmark shows that SemanticLog achieves the best results in five traditional and semantic parsing Metrics (GA 93.3%, PA 93.6%, FTA 84.4%, SPA 83.2%, SPA+ 55.9%). SemanticLog comprehensively surpasses 11 SOTA resolvers including the ChatGPT solution. The semantic parsing accuracy SPA is improved by 18.7% compared to the similar method VALB. The inference speed is better than all LLM resolvers. In the downstream exception detection experiment, fine-granularity semantic tagging increases the detection F1 score by up to 4%. This provides an efficient and reliable open-source solution for the engineering implementation of semantic log parsing in privacy-sensitive scenarios.

Paper address: https://ieeexplore.ieee.org/document/11216353/

③ LogBase: The first semantic log parsing benchmark, enabling AI to truly "understand" every log
Without a good ruler, you cannot measure true progress. The semantic log parsing realm has long faced systematic challenges such as scarce annotations, limited data size, and fragmented evaluation standards. The mainstream benchmark LogHub-2.0 only covers 14 systems and 3,488 templates, which critically restricts the accuracy of AIOps downstream tasks.

The paper "LogBase: A Large-Scale Benchmark for Semantic Log Parsing" (Fudan University & Alibaba Cloud & Tongji University), accepted by ISSTA 2025, builds the first large-scale semantic log parsing benchmark. The benchmark covers 130 open-source projects and provides 85,300 high-quality semantic tagging templates. Compared to LogHub-2.0, the data source size is increased by about 9 times, and the quantity of templates is expanded by 24.5 times. The benchmark is equipped with an 8+16 hierarchical semantic categorization system and an automated building frame GenLog. The benchmark achieves the evaluation paradigm upgrade from syntax parsing to semantic understanding for the first time. A comprehensive evaluation of 15 mainstream resolvers exposes the true shortcomings of existing methods in complex scenarios. This provides a unified standard and reliable foundation for the engineering implementation of semantic log parsing.

Paper address: https://dl.acm.org/doi/10.1145/3728969

Currently, the Alibaba Cloud observability team has integrated the aforementioned innovative technologies into product systems such as Cloud Monitor (CMS), Simple Log Service (SLS), and Application Real-Time Monitoring Service (ARMS). This achieves accurate intelligent alerting, in-depth log understanding, and low-threshold intelligent O&M. This helps enterprises break O&M efficiency bottlenecks, reduce costs, and improve business stability.

The iteration of LLM and AI agent technologies is accelerating. The value of observability data as a key link connecting AI and production systems continues to become prominent. The Alibaba Cloud Observability team will continue to drive technological breakthroughs through academic innovation. The team will improve the operation intelligence technology system, participate in the construction of industry standards, and promote the large-scale implementation of AIOps. This provides more solid artificial intelligence for IT operations support for the digital transformation of enterprises.

Two Thousand Years of Ontology: From Metaphysics to the Engineering Practice of Alibaba Cloud UModel

ObservabilityGuy — Fri, 24 Apr 2026 06:22:45 +0000

This article introduces the evolution of ontology from philosophy to engineering practice, highlighting how Alibaba Cloud UModel utilizes it to unify observability data and empower AIOps.

Have you ever thought that the underlying logic used by Alibaba Cloud Operations and Maintenance (O&M) engineers today to locate server faults is essentially the same as the thinking of ancient Greek philosophers who asked "what the world is made of" more than two thousand years ago? From the first existence analysis frame built by Aristotle in "Metaphysics" to the observability modeling of enterprise Information Technology (IT) systems in the digital age today, ontology has spanned more than two thousand years, gradually evolving from a core branch of metaphysics into the underlying methodology for the digital transformation of various industries. It is never an obscure philosophical speculation in a study. Instead, it always revolves around the simplest proposition: how can we clearly understand the world? How can we turn scattered and personal experiences into a transferable, reusable, and verifiable consensus?

Today, we will follow this path from philosophy to practice to thoroughly analyze the essence of ontology, and see how ontology transforms from an abstract philosophical theory into an engineering implementation tool, and finally completes its native practice in the realm of observability and artificial intelligence for IT operations (AIOps) on Alibaba Cloud UModel.

I. What Exactly is Ontology?
When many people hear about ontology, their first reaction is that ontology is a "profound philosophical concept". However, to put it plainly, ontology is to draw a unified and unambiguous map for the "world" you want to study. The etymology of ontology comes from the Greek words ontos (existence) and logos (doctrine), which literally translates to "the doctrine of existence". In the philosophical system, ontology is the core of metaphysics, and the ultimate questions it needs to answer are: what is the world made of? What is the essence of things? How does existence become existence? Whether it is ontology in philosophy or ontology in the computer realm, the core must solve three problems:

● What truly exists in this world? (What exists?)

● How should we perform categorization and definition for these things? (How to classify?)

● What are the relationships between these things, and how will they interact with each other? (How to relate?)

Here we must distinguish three concepts that are easily confused, which is also the foundation for us to understand the value of ontology:

● Ontology: It defines "what the world itself is". It is the starting point of all cognition. For example, you must first clearly define what a host, pod, and service are, and what the relationships between them are, before you can discuss subsequent O&M operations.

● Epistemology: It answers "how we should understand this world" and is the method of cognition. For example, we need to decide whether to conduct an observation of the Status of a host through Metrics, logs, or traces.

● Methodology: It solves "what means we should use to transform this world" and is the path for implementation. For example, after a fault occurs, we need to determine what steps to take to locate the root cause and complete the disposal.

Without the underlying "map" of ontology, epistemology and methodology become water without a source. If you have not even clearly explained what you want to study, subsequent observations and operations will inevitably fall into chaos. The biggest misunderstanding of ontology is thinking that it is just defining things and attaching labels to things. However, the true soul of ontology is never a static entity definition, but rather dynamic relationships and behaviors. Take the simplest example: to understand "water", we must first clarify that its molecular formula is H₂O. This is the essential definition of water. However, what truly makes us understand "water" is its status changes at different temperatures, its chemical reactions with other substances, and its loop patterns in the ecosystem. Detached from these dynamic behaviors and relationships, "H₂O" is just a cold symbol without any practical significance. This is the essential difference between the static perspective and the dynamic perspective in ontology. The static perspective only focuses on the properties of the things themselves, while the dynamic perspective believes that the essence of a thing can only truly manifest in its relationships with other things and in its own movement and changes. This core cognition is also the fundamental reason why ontology can step out of the philosophical study and take root in the engineering realm. The most painful problem in enterprise digitalization is never "we do not have data", but rather "we have a pile of data, but we do not know what the relationships between the data are, let alone what the business logic behind the data is".

II. Two Thousand Years of Ontology: From Philosophical Speculation to Engineering Practice
The development of ontology has never been a random accumulation of scattered viewpoints, but has completed three key leaps along the main line of "standardizing human cognition".

2.1 Philosophical Foundation: From "Questioning the Origin" to "Building a System"
The starting point of ontology was ancient Greece in the 6th century BC. Before this, people used myths to explain the world. The philosophers of ancient Greece used reason for the first time to start asking "what the origin of the world actually is." Thales said that "water is the origin of all things." He attributed the essence of the world to concrete matter for the first time and opened the prelude to rational inquiry. Heraclitus said that "all things stream, and a person cannot step into the same river twice." He shifted the perspective to "change" and believed that the essence of the world is a procedure of movement. Parmenides proposed that "true existence is eternal and unchanging." The dispute between the two also planted the core proposition of "static and dynamic" in ontology. The person who truly transformed ontology into a complete System was Aristotle. In "Metaphysics", he treated "the study of existence itself" as an independent discipline for the first time. He dismantled the underlying logic of the existence of things using the theory of four causes (material cause, formal cause, efficient cause, and final cause). He also used the ten-category System to perform categorization on all manifestations of existence. Aristotle drew a universal "ontology map" for the world for the first time. This turned scattered inquiries into a reusable Analysis frame.

In the subsequent Middle Ages, European philosophy was incorporated into the theological frame. The dispute between realism and nominalism became the core. Realism believed that universal concepts truly exist. Nominalism believed that only concrete individuals are real, and concepts are just names. This dispute appears to be attached to theology, but this dispute actually clarified the "relationship between concepts and entities." This is exactly the core premise of "knowledge representation" in the computer realm later. From the 17th century to the 19th century, driven by the profound impact of the modern scientific revolution and the rational spirit of the Enlightenment, the modern scientific revolution completely pulled ontology out of theology. Descartes' mind-body dualism separated the cognitive entity and the objective world, and set the research paradigm of "subject-object separation" for modern science. Kant's twelve-category system achieved the conversion of the inquiry of traditional ontology into an epistemological issue. Kant's twelve-category system no longer inquired about the unknowable 'thing-in-itself', but instead studied the a priori logical frame of how humans perceive the world. Hegel's dialectics thoroughly injected the thinking of dynamic evolution into ontology. Hegel's dialectics completed the crucial upgrade from the "description of static existence" to the "description of the laws of motion of existence."

At this point, the philosophical kernel of ontology had become completely mature. The remaining task was to wait for an era that could allow ontology to be implemented.

2.2 Paradigm Transformation: From Philosophical Theory to Engineering Tools
Since the 20th century, the successive explosion of mathematical logic, computer science, and IT has opened the door of engineering for ontology. The development of ontology has also followed the technological wave and completed four crucial steps:

The first step is from text to symbols. The concept of a "universal language" proposed by Leibniz in the 17th century was finally implemented from the end of the 19th century to the 20th century. The mathematical logic founded by Frege and Russell provided ontology with a rigorous and unambiguous formal expression tool. The "existence" that could only be described in words before can now be calculated using symbols and formulas. Ontology transformed from philosophical speculation into a scientific system that can be authenticated and computed.

The second step is from science to the core tool of artificial intelligence (AI). When the discipline of AI was born in the mid-20th century, the first problem to be solved was "how to make machines understand human knowledge." This is exactly what ontology is best at. In 1993, the scholar Gruber proposed the classic definition: An ontology is an explicit specification of a conceptualization. Later, scholars such as Studer performed extension and perfection on this definition, and formed the consensus definition still in use today: Ontology is a formal and explicit normative specification of a shared conceptual system in a certain realm. This completely accomplished the paradigm transformation of ontology. Ontology is no longer a toy for philosophers, but ontology has become the core foundation of knowledge representation in the realm of AI.

The third step is from a single System to the infrastructure of the Internet. Around the year 2000, the Internet rapidly became popular. However, the information on the Internet could only be read by humans. Machines could not understand the information, and the data between different websites were completely isolated islands. The concept of the Semantic Web proposed by Tim Berners-Lee, the father of the World Wide Web, was to use ontology to perform unified semantic tagging on information on the Internet. The publication of standards such as Resource Description Framework (RDF) and Web Ontology Language (OWL) made ontology the underlying infrastructure for knowledge interconnection and interoperability on the Internet.

The fourth step is from the Internet to the era of big data and Large Language Models (LLMs). In 2012, Google published the knowledge graph, and Google took ontology as the "pattern layer" of the knowledge graph. The combination of ontology and Graph Database allowed ontology to achieve large-scale engineering implementation in the era of big data. After the explosion of large language models in 2022, ontology found a new positioning. LLMs have massive knowledge, but LLMs are prone to "talking nonsense", and the procedure is uncontrollable when LLMs infer. However, the structured and precise attributes of ontology can exactly put a "halter" on LLMs, and ontology and LLMs become a Gold combination for the industry implementation of LLMs.

2.3 Modern Exploration: The Breakthroughs and Limitations of Palantir
At this point, many people will certainly ask: Is there an application for ontology in industries such as healthcare, finance, industry, and government affairs? In essence, ontology needs to solve three common difficulties that cannot be bypassed in the digital transformation of all enterprises:

● Data silos. Different systems and departments have different data standards and disconnected semantics. Even though data is available, the data cannot be used together. For example, in the medical industry, the disease glossaries of different hospitals are not unified, and data cannot interoperate at all. In the government realm, the data of different departments is managed independently, and citizens must visit several departments to complete a single task. Ontology builds a unified "translation language" for this heterogeneous data, which allows data from different systems to communicate with each other.

● Experience churn. Most of the core capabilities of an enterprise are hidden in the minds of senior employees. For example, a senior worker in a factory knows what sound a device makes when the device is about to malfunction, and a senior risk control expert in a bank knows what features indicate fraud. Newcomers need to spend several years learning this tacit experience, and if the employees leave, the experience is lost. Ontology can break down this fragmented experience into standardized rules, and turn the experience into reusable and inheritable knowledge in the system. This knowledge will not be lost because of personnel turnover.

● Disconnection between systems and businesses. The IT systems of many enterprises only move offline flows online, but do not incorporate business logic. A heap of data exists in the system, but the data cannot support business decisions, and problems cannot be quickly located when problems occur. Ontology models business entities, relationships, and rules into the system. This makes the system truly understand the business, rather than just storing data.

On the path of modern engineering practice of ontology, Palantir is an unavoidable benchmark. This company can gain a firm foothold in global intelligence, finance, and industrial realms. This is never because of how powerful its big data technology is, but because it is the first to truly implement the core value of ontology into enterprise-level scenarios. Palantir hits the nail on the head by exposing the fatal flaw of traditional data systems. The data of enterprises lies in the database, but the business relationships between data are invisible, and the business experience and judgment rules in the minds of senior employees cannot be incorporated into the system. Everyone is looking at the data, but no one can clearly explain how the business behind the data actually runs. Palantir uses ontology to find the answer to this problem:

● Palantir jumps out of the cold association of primary and foreign keys in traditional databases, and adds business semantics to the relationships between entities. This is not a simple "Identifier (ID) match", but an association with practical significance, such as "Company A holds shares in Company B" and "Account C transfers money to Account D".

● Palantir breaks down the tacit experience in the minds of business experts into configurable and executable rules, and incorporates the rules into the system. This allows the system to replicate the judgment logic of experts.

● Palantir not only records the final status of data, but also traces the end-to-end flow of data generation and circulation. This makes the complete procedure of the business observable and traceable.

This set of strategies allows Palantir to prove the huge value of ontology in highly complex scenarios, such as anti-terrorism, finance risk control, and industrial manufacturing. However, its limitations are also obvious. Palantir is positioned to serve only top-tier customers, takes the route of heavy customization and heavy delivery, has a long implementation cycle, and has an extremely high cost. Ordinary small and medium-sized enterprises cannot afford it at all. Moreover, the threshold for ontology modeling is very high, which requires the cooperation of professional teams and cannot be popularized on a large scale. Palantir has paved the way for the engineering of ontology, but it has also left a new problem. How to turn this system into a reusable, low-threshold, and inclusive capability for the entire industry, so that ordinary enterprises can also use it? This has become a brand-new proposition for the engineering implementation of ontology.

III. UModel: Making Ontology "Light" and "Practical"
If we focus on the observability realm, we will find a deeply meaningful point of convergence. With the continuous deepening of the digital transformation of enterprises, and IT architectures fully evolving toward microservices, cloud-native, and containerization, the core dilemma faced by the observability realm is essentially homologous to the philosophical proposition that ontology sought to solve more than 2,000 years ago. Both aim to solve the fundamental problem of how to clearly define cognitive objects, sort out association relationships, and form a unified consensus. The current enterprise observability systems generally face three major core pain points:

● Data silos and semantic fragmentation. The four major core observability data types, which are metrics, logs, traces, and changes, are scattered in systems of different vendors and different features. The data formats are not unified, and the business semantics are not interoperable. When a fault occurs, O&M engineers need to switch back and forth among multiple platforms for troubleshooting, and cannot achieve end-to-end association analysis or root cause location at all.

● Tacit experience and inheritance failure. Senior O&M engineers can quickly locate faults based on long-accumulated experience. However, this core judgment logic and handling methods exist in the minds of individuals in the form of tacit knowledge. Not only is the training cycle for newcomers long and difficult to master, but the core O&M capabilities of the enterprise cannot achieve standardized accumulation or scaled reuse. The fault handling efficiency always highly depends on individual capabilities.

● LLMs lack a reliable foundation for implementation. The industry generally attempts to apply LLMs to artificial intelligence for IT operations scenarios. However, LLMs lack a standardized knowledge framework in the vertical O&M realm, and have biases in understanding professional terms and business logic. They are highly prone to hallucinate, and their infer procedures and Results are uncontrollable. Therefore, they can never be truly implemented in a production environment.

Alibaba Cloud UModel emerged precisely to systematically solve these industry pain points. Based on the underlying logic of ontology, which prioritizes behavior and takes relationships as the core, UModel creates a universal and unified modeling framework for the observability realm. Essentially, it draws a complete and unambiguous cognitive map of the digital world for complex and heterogeneous IT systems, truly transforming ontology from an abstract theory into a practical tool that O&M engineers can use, know how to use, and afford to use. During the design procedure, UModel is not only an abstraction of data, but also a complete system that integrates data, knowledge, and actions.

UModel Builds a complete product system around four core dimensions. Each dimension not only aligns with the native ideas of ontology, but also forms an irreplaceable differentiated advantage against the industry pain points in the observability realm. This completely distinguishes UModel from general-purpose ontology platforms such as Palantir and traditional observability monitoring tools:

● Standardized semantics definition to solve the core pain points of data silos and semantics fragmentation

With the native ideas of ontology as the core, UModel provides unified and unambiguous standardized definitions for all entities, associate relationships, and business rules in the O&M world. This allows O&M engineers, applications, and AI LLMs to form a consistent understanding of observable data, solving the problem of semantics inconsistency from the root. Unlike general-purpose platforms that have a high threshold of requiring users to Build realm models from scratch, UModel is optimized in depth specifically for IT O&M and cloud Resource Management scenarios. It has built-in mature realm ontology libraries and standardized modeling templates that cover all scenarios such as infrastructure, intermediaries, application performance, and Alibaba Cloud services. Enterprises do not need to build from scratch, and can complete the adaptation of core scenarios out of the box.

● End-to-end closed-loop Build to achieve complete implementation from Data to actions

Based on the graph model, UModel bridges the complete closed loop of "data-knowledge-action". It connects the underlying multi-source observation Data, expert knowledge in the O&M realm, and automated disposal execute actions in depth, achieving end-to-end integration from Data observation and Root Cause Analysis to decision-making and disposal, rather than the simple static data storage and display of traditional tools. At the same time, as the core foundation of Cloud Monitor 2.0, UModel can natively connect to full-stack observability products such as Alibaba Cloud Simple Log Service (SLS) and Application Real-Time Monitoring Service (ARMS). It provides one-stop integration of all observable data, including metrics, logs, traces, and changes. Enterprises do not need to perform complex system integration or custom development, significantly reducing implementation costs.

● Explicit precipitation of implicit experience to achieve standardized inheritance of enterprise O&M capabilities

Closely following the core definition of "rules and constraints" in ontology, UModel dismantles the implicit experience accumulated by O&M engineers during fault judgment, root cause analysis, and emergency disposal into a standardized, configurable, and reusable rule system. This system is precipitated into the system, allowing personal experience to be converted into inheritable digital knowledge assets of the enterprise. Unlike the pattern of Palantir that heavily relies on professional teams to customize rules, UModel relies on visualization modeling tools and standardized modeling flows to completely break the technical barriers of rule precipitation. O&M engineers do not need to master complex ontology theories to independently complete the standardized dismantling of experience and model configuration, achieving universal reuse of core capabilities.

● LLM native integration design to achieve universally beneficial AIOps with bidirectional empowerment

UModel uses a unified ontology model to provide reliable realm knowledge constraints and logical frames for LLMs. This avoids the problem where LLMs hallucinate in vertical O&M scenarios from the root. At the same time, by leveraging the natural language understanding and generate capabilities of LLMs, UModel significantly lowers the technical threshold for ontology modeling and O&M operations, truly achieving bidirectional empowerment between ontology models and LLMs. This is also the core advantage that distinguishes UModel from traditional tools: traditional tools can only achieve simple connection with LLMs, whereas UModel has completed the native integration of the ontology model and the Qwen LLM from the beginning of its design. Users can complete fault localization, root cause analysis, and model configuration through daily conversations. They do not need to memorize complex query syntax or operation instructions, truly achieving universally beneficial "conversational O&M".

In terms of specific architecture implementation, UModel adopts a directed graph structure of "nodes + edges" to completely describe the entire IT world. Each architecture component forms a precise one-to-one mapping with the core concepts of ontology.

At the same time, the implementation procedure of UModel essentially breaks down the philosophical ideas of ontology into standardized flows that can be executed and copied in O&M scenarios. Through five core actions, it helps enterprises convert scattered, tacit O&M experience into standardized ontological models. Each step of the end-to-end flow deeply aligns with the core logic of ontology:

Division of business domains (corresponding to the realm concept definition in ontology). Based on the IT architecture, line-of-business division, and O&M team labor division of an enterprise, clear business domains are delineated, such as the infrastructure domain, application performance domain, Alibaba Cloud service domain, and operational system domain. The border scope and responsible team for each domain are clarified to avoid duplicate model construction from the source, building a foundation for ontological modeling.
Definition of entities and relationships (corresponding to class and association modeling in ontology). The core observability entities within each business domain are sorted out. The properties and field specifications of entity sets, as well as the business semantics relationships between entities, are defined. Examples include the "containment" relationship between a service and a pod, the "running on" relationship between a pod and a host, and the "invoke" relationship between microservices.
Explicitization of O&M rules (corresponding to constraint rule definition in ontology). Through expert interviews and reviews of history fault cases, the tacit experience of senior O&M engineers is extracted and broken down into standardized rule elements. These elements include fault trigger conditions, root cause analysis logic, alerting denoising rules, and automated handling flows. They are then mapped to the constraint rule system of UModel.
Multi-source data fusion (corresponding to instantiate implementation in ontology). Relying on the storage decoupling capability of UModel, it connects with various existing observability data sources of the enterprise. It completes the unified semantics snap of full data, and uniformly maps the metric, log, and trace scattered across different systems into the built ontological model, completely breaking data silos to formlive data that can be subjected to association analysis.
Scenario-based application and iterative optimization. Based on the built ontological model, specific O&M scenarios such as fault early warning, root cause analysis, alerting denoising, and automated handling are implemented. Then, according to the run effects in the actual production environment, the entity definitions, relationship rules, and judgment logic of the model are continuously iterated and optimized. This allows the ontological model to continuously evolve along with the business architecture and O&M requirements of the enterprise.

IV. UModel Practices in Multiple Industries
Based on this set of standardized methodologies, we are also actively exploring the further implementation practices of UModel in industries such as the Internet, finance, industrial manufacturing, and government affairs. This forms replicable implementation solutions adapted to the attributes of different industries, truly authenticating the inclusive value of ontology in the observability realm.

4.1 The Internet Industry: End-to-end Observability of Ultra-large-scale Microservices Models
The Internet industry generally adopts distributed microservices models. Core business traces often span tens to hundreds of microservices, with tens of thousands to hundreds of thousands of container instances running online. The industry generally faces three core challenges. First, observability data is scattered across multiple sets of monitoring tools. Metric, trace, log, and change data lack unified semantics definitions, forming critical data silos. When online faults occur, O&M engineers need to repeatedly troubleshoot across multiple platforms, resulting in extremely low positioning efficiency. Second, core fault handling and root cause analysis experience is highly concentrated in the hands of senior O&M engineers. The parenting epoch for new team members is long, and experience is difficult to standardize for accumulation and reuse. Third, the alert storm problem triggered by massive alerting is prominent. Effective alerting is overwhelmed by invalid information, and fault response efficiency is significantly reduced.

Based on the business architecture and O&M labor division, you can divide five core domains: the application performance domain, infrastructure domain, intermediary domain, Alibaba Cloud service domain, and operational system domain. You can clarify the border and core entity scope of each domain, and build the basic frame for ontological modeling.
You can define core entity sets such as service, instance, pod, edge zone, database, and Microsoft Message Queuing (MSMQ), as well as core semantics relationships such as service invocation, instance deployment, container run, and data read/write. You can build an end-to-end unified ontological model that covers from user requests to infrastructure.
Through reviews of History fault cases and extraction of experience from senior O&M experts, you can break down tacit experience, such as fault root cause judgment, alerting denoising, and automated handling flows, into a standardized rule system. You can accumulate this into UModel to achieve the explicitization and reusability of O&M experience.
You can connect multi-source heterogeneous monitoring data sources. Through UModel, you can complete the semantics snap of full data, achieve the association and connection of end-to-end Data, and completely break data silos.
4.2 The Finance Industry: Compliance-oriented AIOps under IT Application Innovation Transformation
The finance industry is currently in a critical stage of IT application innovation transformation. IT architectures are transforming from traditional centralized architectures to distributed hybrid cloud architectures. Hundreds of operational systems, such as core trading, credit, and wealth management, are running simultaneously in IT application innovation environments and traditional environments. The core pain points of the industry include the following aspects. First, observability data is scattered across monitoring tools of multiple vendors and types. Cross-environment data semantics are disconnected, making troubleshooting extremely difficult. Second, the size of O&M teams is limited, and senior O&M engineers are scarce. Fault handling highly depends on experts, and core experience is difficult to cover all operational systems. Third, the industry faces strict financial regulatory compliance requirements. It needs to achieve end-to-end traceability and auditability of O&M operations and trading traces. Traditional O&M patterns are difficult to meet rigid compliance requirements.
You can combine the IT application innovation transformation architecture and regulatory compliance requirements to divide the architecture into four core domains: the infrastructure domain, core business domain, IT application innovation resource domain, and compliance audit domain. This adapts to the architecture attributes and compliance requirements of the finance industry.
You can define core entities and association relationships, such as hosts, storage, databases, operational systems, and transaction links, to complete the building of the basic ontology model. You do not need to develop from scratch.
You can break down senior O&M experience, such as fault handling of core transaction systems, threat early warning, and compliance audits, into standardized rules, and accumulate them into the ontology model. This achieves system-wide reuse of expert experience.
You can connect multi-source monitoring data between the IT application innovation environment and the traditional environment. You can achieve semantic alignment of cross-environment data through a unified ontology model. This ensures that the end-to-end transaction links are traceable and meets regulatory compliance requirements.
4.3 Industrial Manufacturing Industry: End-to-end Observability of Production Lines in Industrial Internet Scenarios
The discrete manufacturing and process manufacturing industries are accelerating their transformation to the Industrial Internet. A single production line is often equipped with thousands of industrial devices, and the automation rate of production lines continues to increase. The core pain points of the industry include the following. First, the operational data of production line devices, manufacturing execution system (MES) data, and IT O&M data are isolated from each other. They lack a unified semantics definition. Operational technology (OT) and IT data cannot be integrated for analysis. Second, device fault handling highly depends on the personal experience of on-site maintenance personnel. The fault handling cycle is long, which easily causes unplanned downtime of production lines. Third, the core experience in device maintenance and process optimization is scattered across various production bases. When new bases are built or new employees are trained, the experience cannot be quickly reused. Fourth, there is a lack of a standardized predictive maintenance system. Sudden device faults occur frequently, and production continuity is difficult to guarantee.
Centered on the end-to-end production line, you can divide the architecture into four core domains: the device domain, production line domain, process domain, and IT system domain. This covers all scenarios from underlying industrial devices to upper-layer operational systems.
You can define core entities, such as industrial robots, machine tools, sensors, production lines, and process segments. You can also define the ownership relationships between devices and production lines, the transfer relationships between process segments, and the association relationships between device faults and parameters. This helps build a full-scenario ontology model for the production line.
You can extract the experience of senior maintenance personnel across multiple bases in device fault diagnosis, predictive maintenance, and process optimization. You can break down this experience into a standardized rule system and accumulate it into UModel. This achieves cross-base knowledge reuse.
You can connect production line programmable logic controller (PLC) data, sensor data, MES data, and IT O&M data. You can achieve deep integration of OT and IT data through a unified ontology model.
In addition to the standardized O&M scenarios in the aforementioned core industries, UModel explores and implements various innovative scenarios based on the underlying philosophy of ontology, which prioritizes behavior and centers on relationships. This further expands the implementation borders of ontology in the observability realm. These include conversational O&M with native integration of the LLM. Based on the unified realm ontology model built by UModel, the LLM can accurately understand the professional terms, entity relationships, and business rules in O&M scenarios. This fundamentally avoids the hallucination problem of the LLM in vertical O&M scenarios. Users can complete operations such as core system run status queries, fault root cause localization, and O&M policy configurations through natural language. They do not need to master professional query syntax or technical knowledge. This lowers the technical threshold for O&M operations. In response to the common industry status of enterprise hybrid cloud and multicloud deployments, UModel overcomes the limitation of traditional monitoring tools in cross-environment adaptation capabilities. It achieves unified ontology modeling across cloud vendors, deployment environments, and technology stacks. A single ontology model is compatible with the observable data of public clouds, private clouds, and traditional self-managed data centers. You do not need to build independent monitoring or O&M systems for different environments. This reduces the O&M complexity and Management costs under hybrid cloud architectures.

V. Conclusion
More than two thousand years ago, Aristotle wrote Metaphysics to find a unified and unambiguous explanation for the chaotic world. Today, we use UModel to build ontology models for IT Systems. We aim to draw a map for the complex digital world that can be understood and utilized. Today, when the LLM is rapidly popularized, we do not lack AI that can generate Content. What we lack is a knowledge frame that can put a "halter" on AI, make AI truly understand the business, and prevent it from talking nonsense. Ontology is exactly the core of this frame. The combination of the LLM and UModel essentially equips AI with a "business brain." This transforms it from being "eloquent" to being "capable of working and working accurately." This is probably the most charming aspect of ontology. From questioning the origin of the world to locating server faults, ontology has spanned more than two thousand years. What has changed is only the object of research. What remains unchanged is humanity's obsession with "explaining cognition clearly and passing it down." To this day, it still provides the most underlying power for our digital age.

Recommended Reading
🔥 UModel Data Governance: Practice of Building an O&M World Model

🔥 UModel Explorer: Redefining Observability Data Modeling with a Graphical Approach

🔥 From Symptoms to Root Causes: How MetricSet Explorer Reinvents the Metric Analysis Experience

🔥 Building a Unified Entity Search Engine by Using UModel for Observability Scenarios

One Command Equips Your OpenClaw with an X-ray Machine - Alibaba Cloud Observability Makes Farming Lobsters Cheaper and Safer

ObservabilityGuy — Thu, 23 Apr 2026 02:46:10 +0000

One-command observability integration makes OpenClaw AI agent operations transparent via Alibaba Cloud monitoring plugins.
❓Have you experienced this?

However, a problem arises.

The answer for most is: I don't know. 😶

You carefully deployed OpenClaw. However, when these issues arise, you find yourself without the right tools to pinpoint the problem.

This article discusses using one command to equip your OpenClaw with an X-ray machine. This makes every LLM invocation, tool execution, and token consumption visible.

1.What Is Your Lobster Doing? Three “Blind Spots” Are Affecting Your Confidence
📚 Before we start, let's discuss three "blind spots". If you use OpenClaw, at least one has likely troubled you.

Tuning prompts relies on inspiration. Troubleshooting relies on luck. This is not science. It is mysticism. 🎲

Without real-time metric monitoring, you only discover issues after user complaints. By then, a group of users may be affected. ⏰

Both rely on the OpenTelemetry standard protocol. Data is uniformly reported to Cloud Monitor 2.0 of Alibaba Cloud. View and analyze data on the same platform.

Specifically, it records the following types of spans:

These spans have a parent-child relationship. Together, they form a complete trace. You can see a trace view similar to this in the Cloud Monitor 2.0 console:

You can see at a glance how many times the LLM was invoked and how many tokens were used. You can also see which tools were invoked, which step took the longest, and if any errors occurred.

It is that simple to go from "guessing" to "seeing". 👁

The diagnostics-otel uses an event-driven architecture to generate spans. Each event creates a span independently with a different trace ID. It generates the following five types of spans:

● openclaw.model.usage: model invocation (records token usage)

● openclaw.webhook.processed/openclaw.webhook.error: webhook processing

● openclaw.message.processed: message processing (records processing results and duration)

● openclaw.session.stuck: session stuck alerting

There is no trace context propagation between these spans. Simply put, they are just independent data points. The only way to associate them is using business fields such as sessionKey.

Webhook  [openclaw.webhook.processed]  traceId: abc123  
Message  [openclaw.message.processed]  traceId: def456  ❌ Different trace IDs  
Model    [openclaw.model.usage]        traceId: ghi789  ❌ Different trace IDs

enter_openclaw_system              traceId: aaa111  
  └── invoke_agent main            traceId: aaa111  ✅ Same trace ID  
        ├── chat qwen3-235b        traceId: aaa111  ✅ Same trace ID  
        ├── execute_tool search    traceId: aaa111  ✅ Same trace ID  
        └── execute_tool exec      traceId: aaa111  ✅ Same trace ID

In addition to trace integrity, there is a fundamental difference in data richness between the two:

Simply put: The trace from diagnostics-otel is a set of independent "record cards", while the trace from openclaw-cms-plugin is a complete "invocation map". The former only tells you "what happened," while the latter tells you "every step." Use them together. One handles system metrics, and the other handles business traces. They complement each other perfectly. 🤝

3.Setup in One Minute: One-Command Integration Tutorial
🚀 Enough theory. Let's get started. The entire integration process takes less than a minute.

3.1 Get the install command
Log on to the Cloud Monitor 2.0 console. Go to your application monitoring workspace. Choose Integration Center > AI application observability. Click OpenClaw.

In the sidebar, enter the application name and click Click to obtain to generate the integration command immediately. Click the icon in the upper-right corner to copy it with one click.

3.2 Start installation with one command
Open the terminal on the machine where OpenClaw runs. Paste the command you copied and press Enter:

curl -fsSL https://arms-apm-cn-hangzhou-pre.oss-cn-hangzhou.aliyuncs.com/openclaw-cms-plugin/install.sh | bash -s -- \  
  --endpoint "https://Your ARMS-OTLP address" \  
  --x-arms-license-key "Your license key" \  
  --x-arms-project "Your project" \  
  --x-cms-workspace "Your workspace" \  
  --serviceName "Your service name"

Then, sit back and watch it run. ☕

The installation script automatically does the following:

[INFO]  Checking prerequisites...  
[OK]    Node.js v24.14.0  
[OK]    npm 11.9.0  
[OK]    OpenClaw CLI found  
[INFO]  Downloading plugin...  
[OK]    Downloaded  
[INFO]  Extracting...  
[OK]    Extracted  
[INFO]  Installing npm dependencies...  
[OK]    Dependencies installed  
[INFO]  Locating diagnostics-otel extension...  
[OK]    Found diagnostics-otel at: /home/.../extensions/diagnostics-otel  
[OK]    diagnostics-otel dependencies already present  
[INFO]  Updating config...  
[OK]    Config updated  
[INFO]  Restarting OpenClaw gateway...  
[OK]    Gateway restarted  

════════════════════════════════════════════════════  
  ✅ openclaw-cms-plugin installed successfully!  
════════════════════════════════════════════════════

What does it do?

✅ Checks the environment (Node.js, npm, OpenClaw CLI).
✅ Downloads and decompresses openclaw-cms-plugin to the OpenClaw extension folder.
✅ Installs runtime dependencies for the plugin.
✅ Automatically locates the diagnostics-otel extension. If dependencies are missing, it installs them automatically.
✅ Updates the openclaw.json configuration (configurations for both plugins are written at once).
✅ Restarts the gateway to apply the configuration.
You do not need to manually edit any configuration files. The installation script intelligently handles various edge cases. It merges updates into existing configurations instead of overwriting them. It also searches for multiple possible installation locations for diagnostics-otel based on priority.

3.4 Want to uninstall? It is even simpler
If you want to stop using it (though I doubt it), one command does it:

curl -fsSL https://arms-apm-cn-hangzhou-pre.oss-cn-hangzhou.aliyuncs.com/openclaw-cms-plugin/uninstall.sh | bash

The uninstall script automatically cleans up the plugin folder and all related configurations in openclaw.json. It also disables the diagnostics-otel configuration. If you only want to uninstall the trace plugin but keep metrics, add the --keep-metrics parameter.

Clean and quick. No side effects. 🧹

enter_openclaw_system (Request entry: sender and source)
　└── invoke_agent main (Agent execution procedure)
　　　├── chat qwen3-235b  (LLM invoke: model inference + token usage details) 
　　　├── execute_tool search (Tool calling: search)
　　　└── execute_tool exec (Tool calling: code execution)

Each span is annotated with rich properties:

● Duration—see which step is slowest at a glance

● Model information—which model and provider were used

● Token usage—input_tokens, output_tokens, cache_read_tokens, and total_tokens, broken down item by item

● Tool parameters and return values—what tool was invoked, what parameters were passed, and what results were returned

● Error message—displayed in red if an error occurs

What does this mean?

4.2 Token usage breakdown—know exactly where every penny goes
Each LLM span in trace carries complete token usage properties:

Use gen_ai.request.model and gen_ai.provider.name. You can know exactly: which model consumed how many tokens at which step.

Token usage transforms from a "messy account" to a "detailed ledger". 💰

4.3 System running metrics—pulse visible in real-time
Metrics data exported by the diagnostics-otel plugin can build running metric gauges on Cloud Monitor 2.0. This allows real-time monitoring:

● Token usage rate and fee trends — broken down by model and time dimension

● Invoke QPS and response duration — is system throughput normal

● MSMQ depth and wait time — is there a backlog

● Session stall count — Are any lobsters "playing dead"?

● Context size trend — Is the context expanding uncontrollably?

This means:

ENTRY span — A clear "entry point" for the trace

🔗 Session-level association —gen_ai.session.id

📊 gen_ai.span.kind — An AI-specific span categorization system

This standardized approach benefits future data analytics and platform evolution.

5.From Black Box to Transparent: How Observability Changes Your Lobster Farming
📈 Installing an X-ray machine fundamentally changes your "lobster farming" method:

This is not merely an improvement. It is a leap from "blind farming" to "precision farming."

Cloud Monitor 2.0 manages performance and cost, and SLS manages security and compliance. Together, they form a complete control system for the "lobster farm." 🔐

6.FAQs
💡 Here are answers to common questions about the process:

Q: Does the integration impact OpenClaw performance?

Q: Can I install only traces without metrics?

A: Yes. Add the --disable-metrics parameter during installation to skip the diagnostics-otel configuration.

Q: Do traces from diagnostics-otel conflict with traces from openclaw-cms-plugin?

A: The installation script sets diagnostics.otel.traces to false by default. The openclaw-cms-plugin handles trace reporting. They work independently without duplication.

Q: I have configured diagnostics-otel. Will the installation overwrite my configuration?

A: No. The traces, logs, sample rate, and other configurations remain unchanged. It adds necessary fields such as endpoints and headers.

Q: Which OpenClaw versions are supported?

Q: Why is the token consumption always 0?

7.Summary
📋 Back to the first question: Do you know what your lobster is doing underwater?

If the answer is "I don't know", it is time to install an X-ray machine.

The openclaw-cms-plugin + diagnostics-otel, and one command: ten minutes to integrate, bringing three core capabilities to your OpenClaw:

✅Tracing analysis— End-to-end visualization of every LLM invocation, tool execution, and token flow.

✅Real-time metrics— Monitor system pulse in real time, including token consumption rate, invocation QPS, queue depth, and session status.

✅GenAI semantic standards— Standardized data structures. They lay the foundation for cost analysis, performance optimization, and exception detection.

Stop letting your lobster "freestyle" in a black box. Install an X-ray machine. Make every step visible, traceable, and optimizable.

After all, a visible lobster is a good lobster. 🦞✨

❓Interaction time!

Zero-Code Modification in 5 minutes Enables Go Applications to Automatically Obtain End-to-End Observability

ObservabilityGuy — Wed, 22 Apr 2026 02:41:35 +0000

This article introduces the Loongsuite Go agent, a compile-time instrumentation tool that enables zero-code modification for end-to-end observability in Go applications.

💡Are you still worried about the observability transformation of Go applications?
💡Are you still performing manual tracking, modifying code, or importing SDKs?
💡Are you still worried about tracking points affecting performance? Today, we are bringing a solution with zero-code modification-Loongsuite Go agent, allowing your Go application to automatically obtain end-to-end observability capabilities at compile-time!🚀

😫Three Pain Points of Traditional Observability Solutions
In the microservices model, observability has become an essential capability for application O&M. However, traditional observability solutions often face three major pain points:

According to statistics, traditional tracking plans require developers to spend 20-30% of their time on monitoring code, and it is very error-prone.

1.High code intrusiveness
Traditional tracking solutions require developers to manually insert monitoring code into the business code:

// Traditional method: Manual tracking is required.  
func handleRequest(w http.ResponseWriter, r *http. Request) {  
// Manually create a span.  
ctx, span := tracer.Start(r.Context(), "handleRequest")  
defer span.End()  

// Business logic  
result := doSomething()  

// Manually record attributes  
span.SetAttributes(attribute.String("result", result))  
}

This method raises the following issues:

● Code pollution: Business code and monitoring code are mixed together.

● High maintenance costs: The monitoring code must be updated each time the business logic is modified.

● Easy to omit: Developers may forget to add tracking points to some critical paths.

2.Heavy modification workload
For an existing Go application, if you want to integrate observability, you usually need to:

● Import the OpenTelemetry SDK

● Modify each key function and add tracking code

● Configure the exporter and sampling policy.

● Test to verify that the tracking point is correct.

This process can take days or even weeks of workload.

3.Performance overhead concerns
Although runtime tracking is flexible, it incurs certain performance overhead:

● The tracking logic must be executed for each call.

● Serialization, network transmission, and other operations

● may affect application performance.

✨Solution: Automatic Compile-time Instrumentation
The Loongsuite Go agentuses compile-time instrumentation technology to automatically inject monitoring code during the compilation phase, achieving true zero-code modification..

This is an enterprise-level Go application observability solution open-sourced by Alibaba, which has been used on a large scale in the production environment.

Core Strengths
Zero-Code Modification
You only need to add the otel prefix before go build without modifying any business code:

# Traditional method  
go build -o app cmd/app  

# Use the Loongsuite Go agent  
otel go build -o app cmd/app

It is that simple! Your application automatically obtains end-to-end observability capabilities.

🚀Automatic Instrumentation
The tool automatically detects the frameworks and libraries you use and injects the corresponding monitoring code:

● HTTP frameworks: Gin, Echo, Fiber, FastHTTP, and Hertz

● RPC frameworks: gRPC, Dubbo-go, Kitex, and Kratos

● Databases: Database/SQL, GORM, MongoDB, and Elasticsearch

● Caches: go-redis and redigo

● Logstores: Zap, Logrus, Slog, and Zerolog

● AI frameworks: LangChain and Ollama

● More: Supports more than 50 mainstream Go frameworks and libraries.

⚡Performance-friendly
Compile-time instrumentation means:

● Low runtime overhead: Monitoring code is already optimized at compile time.

● No reflection overhead: Does not rely on runtime reflection mechanisms.

● Production-ready: Validated in large-scale production environments.

🎯Case Study: Automatic Instrumentation for the Official MCP SDK
Recently, we implemented automatic instrumentation support for the official Model Context Protocol (MCP) Go SDK. MCP is a protocol introduced by companies such as Google and Anthropic. It is used to integrate LLM applications with external data sources and tools, becoming increasingly important in AI application development.

Why choose MCP?
With the rapid development of AI applications, more and more developers are using the MCP protocol to build LLM applications. However, the observability of MCP applications has always been a challenge:

● Complex protocol: MCP supports multiple operations (such as tools/call, resources/read, and prompts/get).

● Middleware mechanism: The official SDK provides middleware, but users may not actively use it.

● Time measurement: It is necessary to accurately measure the complete time of requests and responses.

Our Solution
We adopted the strategy of automatic injection during initialization. Monitoring middleware is automatically injected when NewServer and NewClient are created, ensuring 100% coverage.

Technical Challenges
The official MCP SDK provides a comprehensive middleware mechanism, but how to automatically inject monitoring middleware without modifying user code is a technical challenge.

Solution
We adopted the strategy of automatic injection during initialization:

// Automatically inject monitoring middleware when NewServer is created.
func afterNewServer(call api.CallContext, s *mcp.Server) {
    if s == nil {
        return
    }
    // Automatically inject monitoring middleware.
    monitoringMiddleware := createServerMonitoringMiddleware()
    s.AddReceivingMiddleware(monitoringMiddleware)
}

// Automatically inject monitoring middleware when NewClient is created.
func afterNewClient(call api.CallContext, c *mcp.Client) {
    if c == nil {
        return
    }
    // Automatically inject monitoring middleware.
    monitoringMiddleware := createClientMonitoringMiddleware()
    c.AddReceivingMiddleware(monitoringMiddleware)
}

Implementation Effect
In this way, we achieved:

100% coverage: The monitoring middleware is automatically injected regardless of whether the user manually invokes AddReceivingMiddleware.
Accurate time measurement: The middleware can be executed before and after request processing, allowing accurate measurement of the complete request-response time.
Automatically record key information:
MCP method names (initialize, tools/call, and resources/read)
Tool name, resource URI, and prompt name
Request parameters and response results
Error messages and duration statistics
Examples
User code does not need to be modified at all:

// User code: Create an MCP server.
server := mcp.NewServer(&mcp.Implementation{
    Name: "my-server",
    Version: "1.0.0",
}, nil)

// Add a tool for normal use.
mcp.AddTool(server, &mcp.Tool{
    Name: "greet",
    Description: "Say hi",
}, handler)

// Run the server.
server.Run(ctx, transport)

After compilation using otel go build, all MCP requests are automatically monitored, including:

● Client invoking tools (tools/call)

● Read resources (resources/read)

● Retrieving prompts (prompts/get)

● Initializing connections (initialize)

Technical Principle: Compile-time Instrumentation
Workflow
The Loongsuite Go agent adds two key phases during compile-time:

Traditional Go compilation flow:
Source code parsing → Type checking → Semantic analysis → Code optimization → Code generation → Linking

Use the Loongsuite Go agent:
Preprocessing → Instrumentation → Source code parsing → Type checking → Semantic analysis → Code optimization → Code generation → Linking

Preprocessing: Analyze dependencies and select applicable instrumentation rules.
Instrumentation: Generate code based on rules and inject the code into the source code.

Core Technologies
● go:linkname: Linking the instrumentation function to the namespace of the target package.

● AST operation: Modify the abstract syntax tree to inject monitoring code.

● Rule-driven: Define instrumentation behavior via JSON rule files.

Instrumentation Methods
Based on framework attributes, we support multiple instrumentation methods:

Intermediary injection (such as MCP and gRPC): Inject the middleware during initialization.
Hook mechanism (such as Redis and Kafka): Utilize the Hook API of the framework.
Direct function peg (such as OpenAI SDK): Instrument directly on key functions.
Struct field injection (such as database and SQL): Inject fields to store metadata.
🚀Get Started in 5 Minutes
Step 1: Install the tool (1 minute)

# Linux/MacOS (Recommended)
sudo curl -fsSL https://cdn.jsdelivr.net/gh/alibaba/loongsuite-go-agent@main/install.sh | sudo bash

# Or download manually
wget https://github.com/alibaba/loongsuite-go-agent/releases/latest/download/otel-linux-amd64
chmod +x otel-linux-amd64
sudo mv otel-linux-amd64 /usr/local/bin/otel

Step 2: Compile the application (1 minute)

# Just prefix the go build with otel.
otel go build -o app cmd/app

Step 3: Configure the export (1 minute)

# Export to Jaeger (development environment)
export OTEL_EXPORTER_JAEGER_ENDPOINT=http://localhost:14268/api/traces

# Or export to OTLP (production environment)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Step 4: Run the application (1 minute)

./app

It's that simple! Your application is now equipped with end-to-end observability capabilities.🎉

Demonstration
After use, you can see the following on Jaeger, Zipkin, or other observability platforms that support OpenTelemetry:

● ✅Complete invocation chain: From HTTP requests to database queries, everything is clear at a glance.

● ✅Detailed performance metrics: duration and error rate of each operation

● ✅Rich contextual information: request parameters, response results, and error messages

Supported export methods
The tool supports multiple export methods. You only need to configure environment variables:

For more information about the configuration options, see Official documentation.

Supported frameworks
The Loongsuite Go agent supports 50+ mainstream Go frameworks and libraries, including:

For more information, see GitHub repository.

⚡Production-grade Performance
Performance advantages brought by compile-time instrumentation:

Benefits:

● ✅Low runtime overhead: Monitoring code is optimized at compile-time, and no runtime reflection is required.

● ✅Production verification: It has been verified in large-scale production environments of companies such as Alibaba.

● ✅Performance-friendly: According to benchmarks, the application performance overhead after instrumentation is usually less than 3%.

💡Note: Although the compile time increases, this only occurs during the developer/build phase and does not affect runtime performance.

Community and Support
Open-source address
● GitHub: https://github.com/alibaba/loongsuite-go-agent

● Document: https://alibaba.github.io/loongsuite-go-agent/

Join the community
● DingTalk group: 102565007776

● GitHub issues: Feedback on questions and suggestions

● Contribution code: Welcome to submit pull requests.

Comparison summary

📚 Related Resources
● 🌟GitHub: https://github.com/alibaba/loongsuite-go-agent

● 📖Document: https://alibaba.github.io/loongsuite-go-agent/

● 💼Commercial edition: https://www.alibabacloud.com/help/arms/application-monitoring/user-guide/monitoring-the-golang-applications/

● 💬DingTalk group: 102565007776

If you find it useful, welcome to star⭐ and share!

References:

● GitHub: https://github.com/alibaba/loongsuite-go-agent

● Document: https://alibaba.github.io/loongsuite-go-agent/

● Commercial edition: https://www.alibabacloud.com/help/arms/application-monitoring/user-guide/monitoring-the-golang-applications/