This article introduces how Chanjet transformed its traditional O&M system into an AI-driven AIOps and observability platform to achieve proactive fault prevention.
1. Background and Challenges
As a leading provider of financial, taxation, and business cloud services for micro and small enterprises in China, Chanjet operates across five major product lines and nine clusters, serving millions of businesses. Currently, its core business has fully completed its SaaS transformation and cloud-native transformation, deploying a multi-tenant, multi-center architecture. With continuous customer growth and growing business complexity, the shortcomings of the original traditional O&M monitoring system became increasingly apparent. These primarily manifested in three major challenges: lack of visibility, unmanageability, and an inability to resolve issues, making a comprehensive system upgrade urgent.
(1) Lack of Visibility: Falling into the "Metric Trap" with Delayed Perception of User Experience
The infrastructure monitoring system for underlying resources like CPU, memory, and disk was established early and highly mature. However, with the rapid development of the SaaS and multi-tenant models, observability tailored to user experience was sorely lacking. Issues directly impacting customer experience—such as domain access anomalies, slow API responses, functional errors, and page freezes—could not be proactively identified by traditional monitoring. Often, troubleshooting only began after customer complaints or public feedback. The team defined this issue as the "Metric Trap": all underlying monitoring metrics appeared normal, yet the end-user experience had significantly degraded.
(2) Unmanageability: Inefficient Alert Floods and Prolonged Troubleshooting
As the system scaled, alert volumes exploded exponentially. A single underlying storage fluctuation could trigger hundreds of correlated alerts, making it difficult for O&M personnel to quickly pinpoint the root cause of the core failure. Meanwhile, insufficient alert tiering and aggregation capabilities meant that critical emergencies were easily drowned out by a flood of low-priority alerts. Previously, it took an average of over 10 minutes from receiving an alert to confirming the root cause. This prolonged the end-to-end failure recovery cycle, leaving significant room for optimization across mean time to identify (MTTI), mean time to know (MTTK), mean time to fix (MTTF), and mean time to verify (MTTV).
(3) Inability to Resolve: Reliance on Experience and Insufficient Proactive Defense
Once a fault was located, emergency mitigation relied heavily on the personal experience of senior O&M staff. Although the team had outlined standardized mitigation strategies such as rate limiting, service degradation, and failover, these were never implemented as one-click automated runbooks within the complex multi-center architecture and multi-tenant environment. Furthermore, proactive risk prevention capabilities were weak. Many failures could have been avoided through proactive inspections and configuration validation, highlighting the urgent need for a systematic prevention mechanism.
In light of these challenges, Chanjet set clear upgrade goals: to increase the overall service level agreement (SLA) from 99.9% to 99.995%, and to build O&M capabilities that enable 99% proactive fault prevention and 10-minute emergency mitigation. Centered on user experience, the ultimate aim was to steadily improve customer satisfaction through a comprehensive reconstruction of the technical architecture and O&M model.
2. Building the Observability System — "Seeing" Issues Clearly and Comprehensively
To address the "lack of visibility" pain point, Chanjet combined business characteristics with application tiering standards (Tier-1, Tier-2, and Tier-3 applications) to build a five-layer integrated monitoring model, establishing layered, comprehensive, and precise observability capabilities.
- Infrastructure monitoring: Covers foundational metrics such as CPU, memory, disk, ports, and network, securing the baseline of system operations.
- Middleware monitoring: Focuses on middleware like Redis, databases, and message queues, monitoring core data such as resource utilization, connection counts, and bandwidth to detect component bottlenecks and connection anomalies at the earliest opportunity.
- Application performance monitoring (APM): Collects operational metrics including GC frequency, thread status, Pod response times, and blocked threads to grasp the real-time health status of applications.
- Business monitoring: Captures business-level error signals from logs, such as database connection exceptions, memory overflows, and rate limiting events, directly addressing fundamental issues in business operations.
- User experience monitoring: Based on access logs for domains and core APIs, it monitors anomalous status codes like 500, 499, and 302, as well as sudden spikes in response latency, evaluating service quality from the user's perspective.
The data collection system of this five-layer model highly aligns with the design philosophy of the Alibaba Cloud cloud-native observability platform (Cloud Monitor 2.0). Cloud Monitor 2.0 integrates several Alibaba Cloud products—Simple Log Service (SLS), Application Real-Time Monitoring Service (ARMS), Cloud Monitor (CMS), and STAROps—into a single unified platform. It provides full-stack, real-time, and non-intrusive data ingestion capabilities, covering multiple data sources such as logs (hundreds of PB/day), metrics (tens of PB/day), traces (trillions of calls/day), events (billions of records/day), containers, and terminals. Utilizing multi-tier hot and cold storage, it reduces the overall cost by 50% compared to open-source self-built solutions at an exabyte (EB) storage scale. Chanjet's five-layer monitoring data is aggregated, stored, queried, and analyzed through this unified platform, providing a data foundation for upper-layer intelligent applications that supports PB-level daily writes and sub-second analysis of hundreds of billions of data points. The monitoring scope strictly matches the application tiers: Tier-3 applications must cover at least the first three layers, Tier-2 applications extend to the fourth layer, and Tier-1 core applications must achieve full coverage across all five layers. The system adheres to three major principles: comprehensive collection, multi-dimensional validation, and timely delivery. Comprehensive data collection eliminates monitoring blind spots; multi-dimensional cross-validation prevents misjudgments from single metrics; and multi-channel notifications via phone, SMS, DingTalk, and email ensure emergency alerts reach on-call personnel immediately.
Building on this foundation, Chanjet introduced the operations digital twin capabilities of Cloud Monitor 2.0 based on UModel to construct a three-dimensional topology architecture of applications, resources, and tenants. UModel organizes entities, relationships, observability data, and O&M knowledge using a unified graph model. It incorporates cloud products, such as Alibaba Cloud Elastic Compute Service (ECS)/Virtual Private Cloud (VPC)/Server Load Balancer (SLB)/ApsaraDB RDS/Container Service for Kubernetes (ACK), Kubernetes resources (Cluster/Pod/Node/Deployment/Service, etc.), applications (microservices/instances/APIs/HTTP/messages/database calls, etc.), and custom enterprise extensions (CMDB, CI/CD pipelines, self-built middleware, O&M SOPs/knowledge bases, etc.) into a unified semantic model. By combining inter-service call chains and gateway distributed tracing, and linking application and tenant tag profiling, O&M personnel can drill down layer by layer from a single user experience alert to quickly locate anomalous instances, resource bottlenecks, and affected tenants. Simultaneously, the system implements refined alert tiering, aggregation and merging, and escalation mechanisms, consolidating alerts from the same fault source into a single event and relying on the root cause analysis module to assist in troubleshooting. The comprehensive observability data and topology capabilities lay a solid data foundation for subsequent AI-driven end-to-end diagnosis and alert noise reduction, enabling end-to-end analysis to be completed within 30 seconds.
3. AIOps Evolution and Platform Architecture — Efficiently "Managing and Curing"
The observability system solved the problem of issue perception, but moving from fault discovery to complete resolution requires continuous iteration of platform capabilities and O&M models. During its cloud-native transformation, Chanjet divided its AIOps evolution into four stages, with each round of upgrades driving exponential improvements in SLA metrics and comprehensive O&M capabilities.
Stage 1: System Construction Phase (SLA 99.9%).
The core action was establishing application lifecycle management, business tiering models, and foundational red-line standards. Rules were set for release changes—internally compared by Chanjet to "red lines, traffic rules, traffic lights, and cameras." Simultaneously, the multi-center architecture and canary release system were implemented to strengthen disaster recovery and release control capabilities. On the platform side, three foundational centers for monitoring, events, and resources were built to unify standards for data collection, templates, and policies. This stage was dominated by manual operations, with the platform serving merely as a support tool, relying on institutional constraints to reduce human errors.
Stage 2: Platform Empowerment Phase (SLA 99.95%).
Efforts focused on three major dimensions: architecture, historical issues, and the full application lifecycle. The team advanced the cloud-native transformation of all businesses to mitigate systemic risks; conducted targeted governance of historical technical debt, such as strong dependencies and outdated components; and applied differentiated O&M strategies based on application stages. Platform capabilities were comprehensively expanded to form a complete architecture comprising the foundational service layer, platform capability layer, and business layer, providing a vehicle for implementing O&M methodologies.
Stage 3: Methodology Solidification Phase (SLA 99.99%).
With systematic standards in place and the integrated DevSecOps/AIOps foundation built on its self-developed cloud stability platform, Chanjet distilled extensive practical experience into a replicable methodological framework—internally known as the "Yonyou Method." Its core is the "0-2-5-10" emergency response methodology: the fault prevention goal is 0 incidents (proactive); timely perception within 2 minutes (MTTI); root cause analysis within 5 minutes (MTTK); and emergency mitigation and recovery within 10 minutes (MTTF + MTTV). The value of this methodology lies in "solidifying" the best practices accumulated in the first two stages into standardized, executable processes. This ensures that any on-call personnel can achieve consistent response quality by following the framework, elevating the team's overall operational capability from "relying on a few experts" to "executable by everyone."
Stage 4: AI Empowerment Phase (SLA 99.995%).
Building on the dual foundation of platformization and methodology, AI capabilities were comprehensively integrated, fundamentally shifting the operational model from "human-led" to "AI-led, human-verified." The platform interaction layer was upgraded to feature an intelligent cockpit (global view), digital employee (AI virtual employee), and personalized workspaces, delivering AI capabilities in a Model Context Protocol (MCP)/skill mode to achieve continuous delivery and capability reuse of AI. Core AIOps modules include: intelligent inspection (prevention), intelligent alerting (alert noise reduction), intelligent diagnosis (boundary localization), intelligent self-healing (multi-dimensional metric linkage), capacity forecasting, and configuration validation. These modules interlock with the monitoring and event centers to form a closed loop of "perception → analysis → decision → execution." AI large language models (LLMs) are deeply embedded in every stage, with the ultimate vision of completely eliminating manual dependency, achieving proactive fault prevention, minimizing human intervention, and rapidly mitigating losses.
The core logic across these four stages is "shifting from relying on humans to gradually relying on platforms/AI tools." First, build systems to lay the foundation; second, build platforms to serve as vehicles; third, distill methodologies to make them replicable; and finally, superimpose AI to achieve autonomy. Behind every improvement in the SLA lies a leap in capability levels and cognitive understanding.
4. Implementing AI Scenarios — Three Major Closed Loops
Under the AI capability framework of the fourth stage, Chanjet implemented several core AIOps scenarios:
Intelligent Inspection — Prevention ("Physical Examination")
Covering comprehensive O&M risks across five major product lines and nine clusters, the system executes three types of automated inspection tasks. The first is O&M risk inspection, which includes regular scanning across dimensions such as resource capacity trends, configuration compliance, dependency health, and component version risks. The second is database red-line scanning, which uses AI to identify high-risk metrics critical to DBAs, such as slow SQL queries, large table bloat, connection pool levels, missing indexes, and large memory issues. The third is change risk identification, which automatically evaluates the impact scope and risk level before executing changes—including SQL anomaly identification, single-tenant abnormal behavior detection, and resource capacity spike prediction.
Issues discovered during inspections automatically generate work orders that are routed to the responsible teams, forming an automated closed loop of "discovery -> work order -> fix -> verification." This replaces the inefficient legacy model that relied on manual inspections and verbal communication. Based on user-defined inspection goals, the system automatically breaks down task steps and continuously executes asynchronous daily and weekly inspection tasks on a scheduled basis. The data platform based on SLS provides high-performance query capabilities for massive data. Combined with UModel, it rapidly executes inspection query jobs, displaying inspection progress and findings. When high-risk issues are encountered, it automatically triggers manual confirmation to ensure that inspection conclusions are reliable and controllable. Intelligent inspection focuses on "data changes," while intelligent validation focuses on "configuration standards"—the two work in tandem to achieve comprehensive prevention.
Fault Self-Healing — Mitigation ("Medical Treatment")
For identified high-frequency fault scenarios, self-healing strategies are predefined and automatically executed. Typical scenarios include: abnormal resource consumption by a single tenant (triggering user isolation); connection pool levels breaching thresholds (triggering API rate limiting); single node unavailability (triggering center failover); degraded downstream service response (triggering service degradation); and sudden traffic spikes (triggering resource scaling).
Once AI detects the early signs of an anomaly, it automatically matches the best scenario and triggers the corresponding recovery action. Currently, an "AI perception + human verification" model is employed: custom scenario anomalies are identified by AI, which recommends a resolution plan; upon manual verification and confirmation, recovery is executed automatically. During this process, the diagnostic and reasoning capabilities of the AIOps assistant provide core support for locating the root cause of the fault. It automatically gathers evidence around alerts, analyzes the impact scope, correlated services, and anomalous metrics, and uses the UModel operations digital twin to reconstruct the full picture and propagation path of the fault, enabling cross-domain correlation analysis rather than isolated troubleshooting. Self-healing capabilities are hosted by the platform's "AI Recognition/Scheduling Center," achieving unified management of fault injection (verification) and fault self-healing (execution) through job orchestration. Newly added fault patterns can be quickly configured into self-healing rules, reducing the reliance on personal experience for damage mitigation. Meanwhile, manual execution channels are retained to ensure fallback capabilities in extreme scenarios.
Capacity Forecasting and Cost Control — Controlling Costs ("Diet Control")
Replacing the traditional capacity alerting model based on static thresholds, AI time-series forecasting capabilities were introduced. In the traditional model, capacity alerts relied on fixed thresholds (e.g., alerting when CPU > 80%), which suffered from unreasonable threshold settings and an inability to predict future trends. In the new model, AI makes comprehensive judgments based on three dimensions: historical capacity patterns (identifying cyclical peaks and valleys), business growth trends (predicting in correlation with business metrics), and sudden event detection (identifying non-cyclical abnormal growth). Cloud Monitor 2.0 provides rich atomic AI analysis capabilities, including operators for time-series forecasting, time-series clustering, and anomaly detection. These operators push computations down to the underlying engine, completing efficient inference over massive volumes of metric data. The CClaw (Chanjet's Agent platform dedicated for micro and small enterprises) automatically generates daily capacity reports, with AI summarizing trends and issuing risk warnings, predicting capacity shortage risks days in advance. Combined with the FinOps cost control module, it achieves integrated management of "proactive capacity risk prediction → resource lifecycle management → cost utilization optimization → resource scoring and cost optimization." The results are twofold: it reduces costs by avoiding resource waste while simultaneously preventing availability failures caused by insufficient capacity.
5. Outcomes and Value
Through four stages of continuous evolution, Chanjet has achieved a comprehensive breakthrough in core metrics:
The overall SLA improved from 99.9% to 99.995%, experiencing a three-step leap in availability (compressing annual downtime from nearly 9 hours to less than half an hour). Fault localization time was slashed from an average of over 10 minutes to under 30 seconds, boosting the efficiency of the MTTK phase within MTTR by more than 20 times. The emergency response system achieved its set goal of "99% proactive prevention and damage mitigation within 10 minutes"—meaning the vast majority of potential faults are intercepted by intelligent inspection and validation mechanisms before reaching users. The O&M model completed a paradigm shift from "human-led, platform-assisted" to "AI-led, human-verified." On-call duty upgraded from a traditional continuous manual monitoring model to a collaborative model of continuous AI monitoring coupled with human handling of escalated events. This shift freed the O&M team's energy from repetitive fault responses, allowing them to pivot toward higher-value work such as architecture optimization and capability building.
From a deeper capability perspective, the most core change is the establishment of a continuous positive feedback loop: "observability data → AI analysis → automated actions → knowledge accumulation." The experience from every fault resolution is accumulated in the knowledge base and fed back to the AI, continuously improving the coverage and accuracy of intelligent diagnosis and intelligent self-healing. The system becomes "smarter" with use, and its reliance on human intervention steadily decreases.
At the same time, AIOps is no longer just a tool for the O&M team; instead, it guides R&D technical transformations through an experience closed loop of "O&M insights → development standards." Performance bottlenecks and architectural risks discovered by AI are translated into red-line rules and best practices on the R&D side, reducing the probability of faults at the source. For example, a specific slow SQL pattern frequently detected by intelligent diagnosis is automatically extracted into a new entry for database development standards, allowing it to be proactively intercepted during code reviews and release pipelines.
From an organizational collaboration perspective, Chanjet achieved systematic accumulation of O&M knowledge on the observability platform. Troubleshooting experience, architectural understanding, and resolution judgments—previously scattered across the minds of various experts—achieved organizational-level knowledge assetization through the digital employee's knowledge base, skill mechanisms, and MCP extensions. With the help of platform tools and AI assistance, newcomers can quickly get up to speed, significantly improving the team's overall response consistency and reliability.
This closed-loop model—"centered on user experience, driven by AI, and integrated from O&M to R&D"—is precisely Chanjet's core methodology for fully stepping into the AI Agent era and building the next generation of technical operations systems. Serving as an Agentic Ops platform, STAROps of Cloud Monitor 2.0 provides an out-of-the-box technical foundation for implementing this methodology through four core capabilities: unified observability data, operations digital twin, AI analysis operators, and a continuous evolution flywheel.
Top comments (0)