This article introduces STAROps' AI-driven log inspection architecture for proactively surfacing unknown anomalies.
For most people, log analysis starts with an alert. An API returns errors, so you search for the request ID. A pod restarts, so you look for the exception stack it printed before exiting. A user reports a failed payment, so you take the account and order number and trace the call path. By that point the problem has already happened, and logs help you reconstruct the scene.
But logs hold far more than the scene of a failure. Application logs record how business logic executes, access logs reflect traffic and service quality, audit logs record who did what and when, and Kubernetes Events record changes in resource state. Developers use them to troubleshoot, security teams use them to audit, and operations teams can use them to spot anomalies in payments, inventory, devices, or user behavior.
What gets overlooked are the changes that have not yet triggered an alert and that nobody has thought to search for: a new version starts sporadically surfacing exception stacks no one has seen before; the overall error rate looks fine, yet one region keeps degrading; a class of high-risk operation appears for the first time outside its authorized window.
On their own, these signals look unremarkable. Left alone, they can also become the starting point of the next large-scale outage.
What Intelligent Log Inspection does is take a look when nobody is asking, and tell us: something here has changed in a way it never did before, and it is worth investigating now.
Rules Catch Known Problems; Inspections Look for Unknown Changes
Rules and alerts are useful. We know which error code means failure and which threshold needs attention, so a single SQL query or alerting rule can monitor it indefinitely. These problems are known knowns: we know the problem, and we know how to find it.
Not every problem in a production environment is that clear.
Sometimes we know something is wrong but not where it is concentrated. The error rate climbs from 1% to 10%, and for the moment we cannot say which service, API, version, or tenant is responsible. That is a known unknown.
Other knowledge lives only in engineers' heads. BizException(code=USER_CANCEL) is a normal business cancellation. A 404 on /healthz can be ignored. ci-bot rotating keys in the small hours is an authorized operation. Without that background, the system keeps reporting all of it as problems. Think of this as the unknown known: the team already knows, the system does not.
The hardest case is the unknown unknown: nobody has defined the problem in advance, and nobody knows what to search for. It might be a new log pattern, a combination of fields that has never occurred before, or a long-tail dimension that has just drifted away from normal.
Intelligent Log Inspection cannot promise to find every unknown problem. What it can do is widen the search: alongside the checks already defined, it keeps looking at new patterns, shifts in distribution, numeric tails, and contradictory signals, and surfaces candidates worth investigating as early as possible.
STAROps' Intelligent Log Inspection
When you hand log inspection to an Agent, the hard part is finding anomalies reliably in a huge volume of data while keeping enough evidence for a human to review. Reading every line is impossible, and sampling a handful of lines is unreliable: low-frequency anomalies slip through, and a few dramatic samples can skew the judgment.
STAROps splits the work into several connected stages:
- Simple Log Service (SLS) performs broad analysis on the data side first
- The LogReduce and dimension drill-down AI operators compress the search space
- The Agent plans its investigation path dynamically from the result of each step
- Confirmed problems enter a watch list for ongoing tracking
- Correlating code, metrics, traces, Skill (a domain skill), and UModel (UnifiedModel, a unified observability modeling framework) extends the investigation across entities and data sources
How SLS Operators and the Agent Divide the Work
In real production, one physical limit cannot be avoided: model context is finite, so you cannot load every log line from an inspection window into it. Naive sampling has its own problems — a low-frequency anomaly may never be sampled, and a few unusual lines can lead the Agent to misjudge the whole picture. Pre-filtering with keywords and fixed SQL only covers the errors you already know about.
SLS provides data-side capabilities such as query and analysis, LogReduce, and dimension drill-down, compressing large volumes of raw logs into structured results a human can review. The STAROps Agent does not replace these capabilities. It generates a plan from the inspection goal, picks time windows, fields, and operators, reviews the results, and then decides whether to drill down further, switch data sources, or stop.
LogReduce extracts stable constants and dynamic variables from indexed text, groups similar logs into patterns (log templates), and then compares pattern counts and variable distributions across windows to find patterns that are new, missing, or spiking. Clustering can also group by fields such as service or module, so the current window can be compared directly against the same period yesterday, the same period last week, or the window before a release.
For example, a mixed set of logs may contain both connection pool exhaustion and downstream call timeouts:
1WARN [worker-17] Connection pool exhausted, tenant=shop-102, wait=812ms2ERROR [gateway] Downstream timeout, provider=bank-a, api=/pay/confirm, elapsed=3201ms3WARN [worker-08] Connection pool exhausted, tenant=shop-518, wait=943ms45Pattern A: <*> WARN [<*>] Connection pool exhausted, tenant=<*>, wait=<*>ms6Pattern B: <*> ERROR [gateway] Downstream timeout, provider=<*>, api=<*>, elapsed=<*>ms7
In a real inspection, what the Agent sees is the overall pattern distribution for the analysis window: how often each pattern occurs now and historically, which ones are new or spiking, and how variable values are distributed. "Connection pool exhausted" does not need to be predefined as a keyword — as long as it forms a new log pattern, the inspection still has a chance to surface it.
Dimension drill-down builds a current window and a historical baseline, then searches dimensions such as service, region，version, and operation — and their combinations — for the ranges with the most significant deviation and the largest contribution, reporting the actual value, the baseline value, and the difference.
This works better than running group by on one field after another. The dimension with the most traffic is not necessarily the one causing the anomaly, and a single field may show nothing at all; the real anomaly may appear only in a combination such as project-a + logstore-x. These two operators detect change and narrow scope; neither is a root cause verdict on its own.
Dynamic Planning and Precise Drill-Down
A log inspection does not hard-code one investigation flow for every system. The first time a task runs, STAROps generates an initial plan from the log structure, the available history, and the inspection goal. Stable checks then repeat on that plan, while the live investigation is decided by the Agent from the evidence in the current round.
One investigation might start with a rising failure ratio, confirm the denominator and the historical baseline, and then call the dimension drill-down AI operator to find the range contributing most. Another might start with a new pattern and then check whether it is concentrated in one version, region, or upstream dependency. When fields cannot explain the change, the Agent turns to clustering the message body. When it suspects a release, it checks versions and changes. When it needs to judge user impact, it goes to requests, traces, or business entities.
The first layer of candidates an operator returns is only an entry point. The Agent verifies that the current window and the historical baseline are comparable, re-queries the actual difference for the leading candidates, and then uses the high-contribution ranges as conditions for the next round of queries. It might first pin down project-a, then compare LogStores and versions only within that project. Each step's result determines what the next step queries, the scope narrows layer by layer, and the query chain is preserved in the Finding (a continuously tracked issue record).
Each inspection round also sets aside room for one bounded piece of autonomous exploration. Even when the existing checks report nothing, the Agent picks one direction to keep looking — unexplained logs, new fields, pattern changes, numeric tails, or contradictory signals. It stops when the evidence already explains the problem, when further queries offer no clear gain, or when the data really is insufficient, and it states the conclusion or the data gap explicitly.
Running Findings Continuously and Improving over Time
A problem found in one inspection should not end up as an isolated notification. STAROps creates a Finding that can be tracked over time, holding a stable identity, its current evidence, the scope of impact, and status changes. When the same class of problem appears again, the system links it to the existing Finding and distinguishes whether it is ongoing, getting worse, recurring, or shrinking in impact.
Notification policy also follows how a Finding changes. A new anomaly, a worsening problem, a recurrence, or a clear expansion in scope can trigger a DingTalk or Lark message; unchanged problems stay in the report instead of interrupting the on-call engineer every cycle.
To start an inspection task, you only need to state the business scope, the run cycle, the priorities, and the notification method. Everything after that improves from real feedback. If BizException(code=USER_CANCEL) is a normal business cancellation, tell the Agent. If health checks, stress test traffic, or authorized operations do not need attention, add the applicable scope. STAROps captures confirmed business semantics and the reasoning behind each judgment as Learning, and uses them to adjust later inspections.
Once a human confirms an unknown problem, it also becomes reusable experience. If a given exception stack has been confirmed to relate to connection pool configuration, a later inspection can reuse the previous drill-down path and review method, first checking whether it is the same problem and then whether it is getting worse. When log fields, business scope, or priorities change, the inspection plan is re-evaluated as well; only validated adjustments become stable checks.
Custom Probes and Extensions
Beyond the checks the system generates, you can add your own probe requirements — for example, watching only fund discrepancies inside the settlement window, checking high-risk operations outside authorized hours, or adding dedicated logic for a particular class of business error code. Custom probes join the inspection plan and run alongside LogReduce, dimension drill-down, and the Agent's own exploration.
Skill and Runbook (a standardized troubleshooting manual) add domain knowledge and investigation methods. After a new Java exception stack shows up, the code MCP (Model Context Protocol) server can retrieve the corresponding file, commits, and published version. When a database connection timeout appears, a dedicated troubleshooting Skill can go on to check the connection pool, slow SQL, and database metrics. Enterprises can also write their own business error codes, authorization windows, handling standards, and system knowledge into their own Skills.
When a single LogStore cannot explain a problem, the Agent uses UModel to locate the related entities and where their data lives, then uses Skills to query metrics, traces, changes, or code. UModel organizes entities such as services, pods, nodes, Agent Threads, and Skills together with their LogSets, MetricSets, TraceSets, and actual storage in one relationship graph. The raw data does not have to be moved into a single store; UModel supplies the object relationships, the data locations, and the field semantics.
That keeps a log inspection from stopping at a single LogStore. The general inspection finds the change first, UModel decides which entities and data the investigation can follow, domain Skills decide how each class of object should be queried, and the code MCP server and other tools gather the evidence — forming a chain of evidence: operators surface candidates, the Agent reviews them step by step, and UModel gathers evidence across sources.
How STAROps Uses Intelligent Log Inspection to Improve Business Reliability
For STAROps, healthy service metrics do not mean the business is reliable. A single task passes through the ingress gateway, authentication, AgentRuntime, Sandbox, tool services, and more; a problem at any one of them can make the task fail, slow it down, or return an incomplete result. We also need to keep watching whether tasks complete, whether performance has shifted, and which Agents, users, and dependency paths the errors cluster around.
So we put STAROps' own Agent Thread logs under routine intelligent inspection. The inspection runs hourly and continuously analyzes request volume, completion status, latency, Agent distribution, and user distribution. When it finds a new error pattern or degradation in a key metric, it follows the Agent and UModel to LLM, tool, and gateway data to confirm what the problem affected and where it occurred.
Starting Inspection from a Critical Business Log
Open the target LogStore in the SLS console and go to the intelligent inspection page to create an inspection task. The inspection requirements can stay generic at first, letting the Agent build an initial plan from the characteristics of the logs; if you already have specific business concerns, you can add them at creation time.
After the first run, the inspection found that log record volume had grown 147% over the previous window, with the increase concentrated in one batch task execution period. The report also included the current window, the comparison window, the actual queries, and a note on uncertainty. Based on the available evidence, the growth looked more like a change in system scheduling behavior than anything that directly affected service health.
The result is well-grounded, but still not closely enough aligned with the STAROps business. Record counts and generic fields alone make it hard to tell which entry point and which users the change came from, or whether it affected real tasks.
Adding Business Semantics to the Inspection Plan
We told STAROps directly: skill indicates which entry-point Skill handled the request, employeeId indicates which user the request came from, and both fields deserve close attention. The Agent updated the inspection plan accordingly, adding Skill request distribution, Skill completion status, Skill performance, user request distribution, and a cross analysis of users against Skills — taking the checklist from 6 items to 11. The next inspection round runs these new checks automatically.
The updated inspection showed its value quickly. In one window, overall P90 latency rose from 94 seconds to 171 seconds, which looked like clear performance degradation. When the Agent broke the numbers down by Skill, however, no individual Skill's P90 had degraded; what actually changed was traffic composition, as Skills with a higher share of slow requests accounted for more of the current window. The final report classified it as a change in traffic composition rather than a drop in service performance.
As you can see, user feedback changed how the Agent read the logs, and with it the subsequent checks and drill-down paths. Confirmed business semantics carry into later inspection plans, so nothing has to be explained again each round.
Running Continuously, Notifying on Problems
The inspection task needs to run hourly, but notifications do not need to go out every hour. We added problem trigger conditions to the task: an error rate above 5%, or a new significant finding. STAROps then breaks that down itself: it pushes to the "STAROps Alerts" DingTalk bot only when a new error type appears, when a Skill's non-completion rate is markedly high, when P90 clearly degrades, or when errors concentrate in a small number of users.
In a later inspection, the non-completion rate rose from 1.01% in the previous window to 6.2%, crossing the notification threshold. The notification the bot received carried the error distribution by Skill and by user, performance metrics, and an initial evaluation; the problem had already narrowed to context canceled clustered on one Skill path within a specific time period. Normal inspections that trigger nothing still produce a report, but they do not repeatedly interrupt anyone.
Widening the Investigation with Skills and UModel
A single LogStore can tell us what happened in an Agent Thread, but not necessarily why. We went further and linked this inspection to STAROps' own Skills and UModel: an Agent Thread can be correlated to its entry-point Skill, user, LLM calls, tools, and gateway, and each class of entity has its own LogSet, MetricSet, or other data sources.
When Thread logs show an abnormal non-completion rate, the Agent can follow UModel to check the related entities. In one cross-source inspection, the failure ratio on Threads was still 0%, while the correlated LLM call data showed clustered 429 rate limiting; some gateway SSE statistics and /metrics requests had already been identified as known noise. The final report split the evidence into P1 normal, P2 abnormal, and P3 known noise, rather than lumping every red metric together.
The STAROps reliability inspection began with a single LogStore. The first round was not closely enough aligned with the business, so we added business semantics; the check plan updated accordingly, and the notification policy moved from "send every round" to "notify on problems." Skills and UModel then extended the investigation to more entities and data sources. As the system and the business keep changing, this inspection will keep being calibrated, becoming a continuously running reliability safeguard.
Conclusion
We used to open the logs only after something had gone wrong. Intelligent Log Inspection moves that one step earlier: the Agent continuously watches for pattern and distribution changes in logs, and looks for signals worth investigating even when nobody is asking and no rule exists yet.
A useful inspection task does not need every rule written on day one. It can start from one LogStore and a generic plan, then use its own results to fill in business semantics, adjust the check plan and the notification policy, and finally reach more entities and data sources through Skills and UModel.
Round after round of inspections and feedback turn unknown problems into reusable experience, retire known noise from notifications, and keep tracking the problems that genuinely changed. Problems surface earlier, investigations narrow faster, and every notification that does go out is worth a closer look.
Top comments (0)