Every on-call team climbs the same ladder. The agent at the top is only as good as the rungs underneath it.
At 2:07 a.m. the page says checkout is failing, and the investigation starts wherever the telemetry happens to live: a log search in one tool, a latency chart in another, no traces, and anything older than a few weeks in an archive nobody can query. Agents are about to take on much of that investigating. Gartner expects half of organizations to run IT operations with AI agents and minimal human involvement by 2030, up from almost none today (Gartner). Dropped into a scattered stack, an agent inherits the scatter. Getting from log lines to agent investigations takes five rungs, each resting on the one below, and Amazon OpenSearch Service with its observability stack covers all five on one set of data.
The climb starts with log lines
On almost every logging domain I look at, the first rung is the same: log search on tiered storage. Applications write logs, a shipper such as Fluent Bit sends them to an Amazon OpenSearch Service domain, and engineers read them in dashboards. Hot storage on the data nodes is the fast, expensive part, so the last few days live there and older indexes move to the UltraWarm tier. UltraWarm keeps those indexes in Amazon S3 with caching in front, charges only for the data you store, and answers queries through the same APIs, though warm indexes are read-only and broad searches across many of them run slower. Data past the warm window often leaves the domain for an S3 archive that nobody queries during an incident. When the page comes, the questions arrive in a predictable order: which service is erroring, which request failed, which customer it hit. Logs answer the first question well. The second and third are harder, because nothing ties a log line to the request that produced it, and the metrics that showed the spike live in a different tool that names the same service differently.
With agents everywhere in the news, the tempting move from here is straight to an agent. The step that really changes the investigation is quieter, and it is correlation. On the second rung, OpenTelemetry gives every signal the same identifiers and the same attribute names. A trace ID ties a log line, a span, and a metric exemplar to one request, and the semantic conventions mean that service.name refers to the same service everywhere it appears. Instrumenting once with the OpenTelemetry SDKs or auto-instrumentation, then pointing a stock OpenTelemetry Collector at an Amazon OpenSearch Ingestion endpoint, gives one pipeline that routes by signal. Logs and traces land in OpenSearch Service, metrics go to Amazon Managed Service for Prometheus, and the service map is built from the traces in flight.
Once that pipeline exists, an investigation in OpenSearch UI follows a path instead of a hunch. From the alert, the rate, error, and duration view shows which service is failing, the service map shows the red edge from checkout to payment, the trace waterfall opens the failing span, and its logs come up with the trace context already applied. One PPL query gives the overview across every service:
source = otel-v1-apm-span-*
| where status.code = 2
| stats count() by serviceName
| sort - count()
Without that correlation, an agent reasoning over logs and metrics that share no identifiers has to guess which events belong together. Its answer sounds just as confident when the guess is wrong. Correlated, consistently structured telemetry is the precondition for every rung above this one.
Agents join the pipeline
With correlated signals in place, agents join the pipeline in two roles, and each role is a rung. On the third rung, the agent is a workload: one more OpenTelemetry producer. Each model call and tool call is a span carrying gen_ai.* attributes for the model, the operation, and the token counts, so its traces sit beside the services it calls, as Read Your AI Agent's Mind and Implement Agent Observability describe. On the fourth rung, the agent becomes the investigator and the first reader of the operational data. It reaches OpenSearch Service through MCP and runs the same queries an engineer would, and Amazon OpenSearch Service MCP Apps return the trace waterfall and service map in the chat thread so a person can check its work, the subject of Stop Switching Tabs. OpenSearch UI also includes a built-in Investigation Agent that plans its queries, reflects on what comes back, and returns ranked root-cause hypotheses with its reasoning visible.
Investigate the investigator
Between investigating and acting, trust drops off. In a Grafana Labs survey with ETR, 85% of IT leaders said they were comfortable with AI running read-only investigations, while only 52% were comfortable with AI fixing incidents on its own (Grafana Labs, 2026). Closing that gap takes evidence about the agent itself.
On the fifth rung, at the top of the ladder, the investigator is also something to investigate. Every query it runs and every model call it makes emits spans into the same pipeline, so its work appears in OpenSearch UI like any other service. Where strace shows each system call a process made, the investigator's own trace shows each query it ran, what came back, how many tokens it spent, and how long each step took. When an agent proposes a root cause, the evidence it actually retrieved sits beside the evidence it describes.
Before an agent changes anything in production, four controls need to be in place. Every AI query should be audited, so there is a record of what the agent asked and what it saw. Identity has to travel end to end, so the agent's access through MCP is scoped by IAM and enforced by fine-grained access control rather than granted broadly. Sensitive fields should be masked before an agent or an engineer reads them. And each investigation needs a budget for tokens, cost, and runtime, because an agent stuck in a retry loop burns money quietly. With those controls in place, an agent can investigate freely while a person approves the action, which is the right place to draw the line today.
Bringing it together
Five rungs carry an on-call practice from log lines to agent investigations: log search on tiered storage, correlated signals in one pipeline, agents observed as a workload, agents working as investigators, and investigators that can be audited. Correlation, on the second rung, decides whether an agent's confident answer is also a correct one, which makes it the rung not to skip. Amazon OpenSearch Service holds the whole climb on one set of data, from OpenTelemetry ingestion through OpenSearch UI to MCP, so each rung builds on the last instead of adding another tool. A future post zooms in on one rung and shows how to compute what agents cost, call by call, from the traces they already emit.
Top comments (0)