Toward Production-Grade AgenticOps: How STAROps Builds Generalizable Root Cause Analysis

This article introduces STAROps, an AgenticOps framework for reliable, generalizable root cause analysis in production.

When people evaluate AgenticOps, they often start with the obvious questions: How many tools can the agent use? Can it run automated inspections, propose a fix, or even carry out a change?

But as we built STAROps, we kept coming back to a more fundamental question: Does it identify the right root cause?

We see root cause analysis (RCA) as the core of AgenticOps because it determines where every subsequent step is directed and what that step does. If the root cause is wrong, impact evaluation, remediation, change execution, and outcome validation all focus on the wrong target. When an agent can only advise, a wrong RCA wastes engineers' time. When an agent has permission to act, a wrong RCA becomes a production risk.

RCA sets the direction. Every other capability determines only how quickly and how far the agent can follow it.

When the Root Cause Is Wrong, Every Action That Follows Is Wrong

An alert usually tells us which metric crossed a threshold, not where the failure began.

A frontend endpoint returning 5xx errors may point to a problem in the frontend code, but it could also be caused by a slow downstream database, an unavailable cache, exhausted node resources, or a recent configuration change. High CPU is not necessarily the root cause either: it may be the result of a traffic spike, or it may coincide with a full garbage collection cycle, thread pool exhaustion, or a runaway loop.

Those distinctions change the response. Mistake a node resource problem for insufficient application capacity, and you may keep scaling out pods that run on the faulty node. Mistake a slow SQL query for a frontend code defect, and you may trigger a rollback that changes nothing. Mistake a traffic drop caused by an accidentally scaled-down Deployment for a network problem, and the investigation starts in the wrong direction.

General-purpose agents can already query a wide range of data and produce well-structured reports that sound plausible. But a complete report does not prove that the agent found the object that actually failed. If it stops at the entity where the alert fired, or mistakes a pronounced anomaly along the propagation path for the root cause, more elaborate reasoning can make the wrong conclusion even more convincing.

Why a Proof of Concept Cannot Prove RCA Capability

When teams validate RCA, they usually choose a handful of known faults: a Pod stuck in a crash loop, a slow SQL query, or a Redis outage. They then check whether the system returns the expected answer.

This is a reasonable way to validate a proof of concept. The scenarios are well defined and the results are easy to verify, so teams can quickly confirm that the data, tools, and product workflow operate end to end. But it proves only that the system handles scenarios A, B, and C. It does not show that the system will know where to look when fault type D appears.

If every alert type is mapped to a Skill or workflow, a proof of concept will usually perform well. The alert type, investigation entry point, query sequence, and expected answer are all specified in advance, so the agent only has to follow a path that someone has already designed. Change the alert, or let the same fault surface at a different point in the business, and that path may no longer work.

That does not mean Skills lack value. Experienced site reliability engineers (SREs) know which metrics to check first when latency rises, when a change must be examined, and which symptoms tend to send an investigation down the wrong path. That experience should be captured. Skills work best as investigation leads and checklists that help the agent avoid dead ends; they should not define the limits of RCA.

Real Production Faults Follow No Fixed Playbook

In production, however, far more things can go wrong. A problem may originate in an application, container, node, database, cache, or network. It may also follow a release, a scaling operation, or a configuration change. Topologies keep changing, data may be incomplete, and several anomalies can appear in the same time window.

The same alert can have entirely different causes, and the same root cause can trigger different alerts in different systems. The root cause may be in a database while the alert fires on the frontend. It may be on a node while the first user-visible symptom is a traffic drop on a business endpoint. Real faults routinely cross data domains and propagate through call, deployment, and host relationships.

Generalization does not mean building a model that knows the answer to every fault. A more realistic standard is that, when no workflow matches exactly, the agent still knows how to begin. It first identifies the entity that the alert refers to, then follows relationships to the relevant services and resources. When several candidates emerge, it gathers more evidence, checks the sequence of events, and eliminates hypotheses that cannot explain the symptoms. If the evidence is insufficient, it knows when to stop instead of forcing a root cause.

Fault scenarios cannot be exhaustively enumerated, but the basic method of investigating them can be reused. STAROps is designed to keep an investigation moving and converge on an answer even when the fault is unfamiliar.

How STAROps Turns RCA into a System Capability

Because a scenario catalog can never cover every production fault, simply adding more workflows cannot be the primary goal. STAROps focuses on the investigation itself: how the agent understands the current system, decides what to examine next, demonstrates that a conclusion is reliable, and learns from failures in production.

First, Understand the System: UModel

Many RCA investigations go wrong at the first step because different systems use different names for the same object.

In application performance management (APM), a business service appears as a service. In distributed traces, it appears as a set of spans. In Kubernetes, it maps to a Deployment and its pods; further down, it maps to an ECS instance or a node. Logs, metrics, and change records use different fields as well. Without a unified model of these object relationships, the agent has to match objects using names and context. It can easily split one object into several or mistake a neighboring object for the root cause.

UModel organizes these objects and relationships. It records which pods run a service, which Deployment each pod belongs to, which node hosts it, which databases the service depends on, and where to query the relevant metrics, logs, and traces.

The immediate benefit is that the agent knows exactly what it is investigating. Whether the investigation moves down from a business endpoint to a service, pod, and node, or outward from a change to the affected services, it can follow explicit relationships. The model proposes hypotheses and interprets evidence; UModel provides a stable map of the system.

Static Topology Is Not Enough: Map the Investigation, Too

A console topology usually shows which objects exist and which ones depend on others. A fault investigation needs to capture something else as well: which objects have already been checked, where anomalies were found, which candidates are under suspicion, and which branches remain unexplained.

On top of UModel, STAROps maintains a dynamic investigation topology. The investigation starts with the alerting entity and expands as evidence accumulates. When a candidate is ruled out, that branch stops. If several pods on the same node degrade at the same time, the investigation converges on the host. If database connections slow down, the agent checks the SQL queries, connection pool, and upstream and downstream traces.

This graph serves as both an investigation map and a record of the process. Metrics, logs, traces, events, and changes are attached to the relevant objects. Candidate root causes are retained along with the evidence for and against each one. Engineers can see why the agent continued in a particular direction, add information at critical points, or take over the investigation.

Use a Benchmark to Expose Answers That Only Look Right

RCA agents face a tricky problem: an answer can be wrong even when the explanation sounds convincing. From the final report alone, it is often hard to tell whether the agent followed the evidence to the root cause or guessed a common answer from the alert's surface symptoms.

That is why RCA-Bench records more than a single root cause label. The public RCA-100 dataset contains 103 fault cases across six major categories and 28 fault types. Each case records the root cause entity, fault type, propagation path, and key evidence. In a queryable environment, the agent must decide for itself what to examine first and where to go next.

We evaluate three things separately: whether the agent found the right root cause entity, identified the right fault type, and supported its investigation with evidence. The first two are scored mainly from entity relationships and fault-type relationships in UModel. Deterministic rules currently account for about 82% of the overall score. The LLM is used only to help judge the direction of the investigation and whether the evidence is sufficient, so one model does not have complete authority over another model's score.

Production Failures Must Feed the Next Iteration

An offline benchmark can show whether a capability exists, but production introduces permissions, data integrations, customer-specific topologies, and version differences. A version that scores well offline may still fail in production because it cannot retrieve the data, chooses the wrong tool, or converges too early.

STAROps retains the task input, tool calls, query results, errors, evidence, execution time, and user feedback from production runs. For a low-scoring run or failure case, we first determine whether the problem lies in object identification, data retrieval, investigation direction, evidence quality, or the final conclusion.

Reproducible failures become regression cases. Whenever UModel, a tool, a Skill, a model, or an investigation strategy changes, we validate the change against those cases. This turns a one-off fix into a regression check that every later version must pass.

Head-to-Head Results: A Complete Report Does Not Mean an Accurate Root Cause

The published RCA agent benchmark uses a stratified evaluation set of 30 cases drawn from RCA-100. STAROps and OpenClaw + DeepSeek-V4-Pro receive the same task inputs and are evaluated with the same brise scorer.

Evaluation dimension	STAROps	OpenClaw + DeepSeek-V4-Pro	Difference
Overall score	75.23	51.02	+24.2
Root cause entity	90.	52.8	+37.2
Fault type	58.	33.3	+24.7
Investigation process	75	69.8	+5.2

The totals matter, but the breakdown matters more.

The gap in the investigation-process component is only 0.052, equivalent to 5.2 points in the table. This shows that a general-purpose agent can already perform multiple rounds of queries and produce a reasonably complete analysis. The real separation comes from identifying the root cause entity and the fault type. Choose the wrong object, and every subsequent metric, log, and trace centers on that object. Misclassify the fault type, and the remediation recommendation loses its foundation.

By major fault category, STAROps leads by 50.6, 29.9, 27.6, and 24.2 points in database, node, code, and resource cases, respectively. It trails the comparison system by 11.2 points in traffic cases, mainly because of rate-limiting scenarios. Rate limiting, DNS, CDN, third-party services, and public network paths are the areas STAROps will focus on next.

Case Studies

Hard cases do not necessarily involve obscure technologies. The real difficulty is often that the alert fires far from the root cause while several plausible explanations appear at the same time. The following two cases fit that pattern.

product-catalog Traffic Drops, but the Root Cause Is an Underlying Node

The high-node-CPU case begins with a traffic-drop alert for product-catalog::ListProducts.

If you look only at the business-level data, frontend is the most obvious suspect. Its traffic and error rate change at the same time, and it is the upstream entry point for product-catalog. The general-purpose ReAct agent concluded that the load balancer had failed, while OpenClaw attributed the problem to frontend HTTP 5xx errors. Both reports cite data and describe a propagation path that sounds plausible.

The actual root cause is Kubernetes node cn-hongkong.10.0.1.107. CPU utilization on that node rose from about 10.38% to 99.98%, affecting multiple pods on the same node. frontend traffic dropped by 66.66%, followed by a 62.48% drop in ListProducts traffic, which ultimately triggered the business alert.

STAROps did not stop at frontend. It moved from the alerting operation to its service, then followed the runs-on relationship to the pod and host node. Other nodes remained healthy while several pods on this node degraded at the same time. Together, those two pieces of evidence narrowed the investigation to the infrastructure layer.

In this evaluation, ReAct scored 15, OpenClaw scored 12, and STAROps scored 94. If the team had acted on either of the first two conclusions, it might have kept investigating the load balancer or scaled out frontend while leaving the node with saturated CPU untouched.

Node CPU stays near 100% 
↓ 
Multiple pods on the same node are affected 
↓ 
frontend and other business services degrade at the same time 
↓ 
product-catalog API traffic drops, triggering an alert

Frontend Checkout Slows Down, but the Root Cause Is a Slow SQL Query

The slow-SQL case begins with an alert on frontend::POST /api/checkout. At the same time, the environment shows changes in checkout latency, Kafka lag, Java Virtual Machine (JVM) garbage collection, thread count, and Deployment scaling. Any one of these signals, viewed in isolation, could look like a plausible root cause.

The general-purpose ReAct agent concluded that checkout contained a code defect, while OpenClaw located the problem in frontend's checkout logic. Both captured part of the anomaly, but their investigations stopped midway along the propagation path.

STAROps continued downstream through the traces. checkout depends on cart, which in turn depends on inventory. A slow inventory trace showed a SELECT query taking about 10.8 seconds, plus another 2.1 seconds to acquire a database connection. The slow query held up inventory, and the delay then propagated through inventory → cart → checkout → frontend, ultimately appearing as a slow response from the frontend endpoint.

There were several distracting signals along this path. Kafka lag was measured only in milliseconds and could not explain a timeout measured in seconds. The garbage collection, thread-count changes, and Deployment scaling occurred after the services had begun to degrade, making them more likely consequences of the slow query. Only after the signals were placed on a single timeline did the evidence converge on the slow SQL query in inventory.

ReAct and OpenClaw both scored 15 on this case, while STAROps scored 84. If the team had treated the problem as a code defect, it might have rolled back frontend or checkout while leaving the SQL query that was blocking the call chain in place.

inventory slow query and blocked connection acquisition 
↓ 
cart calls to inventory slow down 
↓ 
checkout call chain latency increases 
↓ 
frontend checkout slows down, triggering an alert

Conclusion

One of these faults occurred at the node-resource layer; the other occurred along the database access path. They have little in common on the surface, and STAROps used different data and investigation paths for each. But the method was the same: identify the object, then gather evidence along its relationships. When multiple candidates emerge, place them on a single timeline and determine which one explains the full propagation path and which ones are merely accompanying symptoms.

This is what generalization must solve. We cannot make an agent know every possible fault in advance. We can, however, ensure that it can move an investigation forward when the problem is unfamiliar and stop at the right boundary when the evidence is insufficient.

That is why RCA sits at the core of STAROps. Impact analysis, remediation recommendations, and change execution have a reliable starting point only when the agent consistently identifies the right object and cause. As agents gain more permission to act in production, the cost of a wrong root cause rises. A single misjudgment can escalate from misleading an engineer to triggering the wrong scale-out, rollback, or configuration change.

We expect the capability gap in AgenticOps to center increasingly on judgment quality. Tool calling and automated execution will become baseline capabilities. What will determine whether an agent is truly ready for production is whether it can build a complete evidence chain for an unfamiliar fault, distinguish the root cause from accompanying symptoms, and stop when the evidence runs out.

For STAROps, root cause identification is an engineering discipline that must be continuously calibrated against real faults. Keeping its judgments accurate and its limits explicit remains our most important task as we move AgenticOps into production.