The Blue Mosque in Istanbul, photo by Diana Todea
Observability has never suffered from a lack of data. We collect metrics, logs, traces, events, alerts, and increasingly more context about the systems we operate. The harder problem is using all of that information effectively when something goes wrong.
When an application starts behaving differently, an engineer may need to move between several tools, write multiple queries, correlate signals manually, inspect dashboards, and decide which deviations actually matter. Ultimately, the questions we are trying to answer are much simpler than the process: What changed? Why did it change? What should I investigate next?
This is where AI-assisted observability becomes interesting. Not because an LLM can magically understand a production system, but because we can give an AI agent controlled access to the same tools and data engineers already use.
Model Context Protocol (MCP) provides one way of doing that. But MCP is only one piece of the puzzle. To build something genuinely useful for troubleshooting, we need to combine access to live observability systems with anomaly detection and repeatable operational knowledge.
Start with what developers actually need
Before adding AI to an observability stack, it is worth asking what problem we are actually trying to solve.
Most engineers don’t want another observability interface. They want to understand their systems faster. That usually means having visibility across telemetry, detecting unusual behavior earlier, reducing mean time to resolution (MTTR), making troubleshooting more efficient, and being able to trust the data used to make decisions.
Consider a fairly normal incident. An alert fires because an application’s error rate has increased. An engineer opens a dashboard, checks the relevant metrics, changes the time range, looks for related logs, follows a trace, perhaps checks another service, and gradually builds a hypothesis about what happened.
None of those steps is particularly revolutionary. The difficult part is connecting them. Engineers repeatedly move between tools and reconstruct context while deciding what to investigate next.
An AI agent can potentially coordinate some of this work. But before it can investigate anything, it needs access to the system.
What MCP changes
The Model Context Protocol provides a standardized way for AI applications to connect to external tools and data sources. In an observability environment, an MCP server can expose capabilities such as querying telemetry, retrieving information from an API, searching documentation, checking configurations, or interacting with an anomaly detection system.
The important distinction is that connecting an agent through MCP does not mean the model suddenly contains your observability data. The model can decide that it needs information, invoke an available tool, receive the result, and then use that result as additional context.
This matters enormously for observability because production data is inherently dynamic. Yesterday’s CPU usage or error rate is not particularly useful when I ask why a service is behaving differently right now. The agent needs a way to retrieve the current state of the system.
That is the role MCP plays. In my talk, entitled “MCPs for Observability Stacks”, I described it as giving the agent “hands and eyes”: it provides a standardized connector to live tools, APIs, documentation, and data. This also comes with familiar operational concerns, including authentication, authorization, timeouts, compatibility, and careful scoping of the tools an agent is allowed to invoke.
The key point is that MCP provides access; it does not provide expertise.
Giving an agent tools doesn’t make it an SRE
Suppose we connect an AI agent to our metrics, logs, traces, anomaly detection system, and documentation. The agent now has considerably more information available to it, but it still has to decide what to do with that information.
Which signal should it inspect first? What time range should it use? Which query is appropriate? When should it move from metrics to logs or traces? What evidence should it collect before suggesting a possible explanation?
An LLM can improvise these steps, but improvisation is not necessarily what we want during an incident. Operational teams already have troubleshooting practices, runbooks, query patterns, and accumulated knowledge about how particular systems should be investigated.
Rather than expecting the model to reinvent those practices every time, we can give it a reusable workflow. This is where skills become useful.
MCP provides access. Skills provide the workflow.
A skill is a reusable set of instructions that teaches an AI agent how to accomplish a particular task. It can contain domain knowledge, recommended steps, examples, and guardrails. In that sense, a skill can capture some of the operational knowledge that would otherwise live in an engineer’s head, a runbook, or scattered documentation.
Imagine that we want an agent to investigate a latency regression. A skill could instruct it to first establish the affected service and time period, inspect latency and error signals, look for unusual behavior, and then use related logs or traces to gather more context. MCP provides the agent with the actual tools required to perform those actions against the live system.
This distinction is important because skills and MCP solve different problems. A skill describes how a task should be performed, while MCP gives the agent access to the systems required to perform it.
Skills provide expertise while MCP provides access. A skill tells the agent which workflow to follow and why; MCP provides the live capabilities needed to execute that workflow.
There is still another question, though. If our observability system contains thousands or millions of time series, where should an investigation begin?
Where anomaly detection fits
Static alert thresholds remain extremely useful, but real systems don’t always behave according to simple thresholds. A metric can remain within an acceptable absolute range while still behaving very differently from its normal pattern.
In time-series data, an anomaly is a data point or pattern that deviates significantly from expected behavior. It might appear as a sudden spike or dip, a change in trend, or another unusual pattern that cannot be explained by normal fluctuations in the data.
Anomaly detection attempts to identify these deviations automatically so that unusual behavior can be surfaced for further investigation.
This gives us an interesting starting point for AI-assisted observability. Instead of asking an agent to blindly explore a large amount of telemetry, anomaly detection can identify where something unusual is happening. The agent can then use the observability stack to investigate that behavior further.
In other words, anomaly detection helps narrow the search space. It doesn’t tell us everything that happened, but it can tell us where to start looking.
An anomaly is a signal, not a root cause
This distinction is particularly important when introducing AI into the workflow.
Suppose an anomaly detector identifies an unusual increase in application errors. It has found something statistically unusual, but it has not necessarily identified an incident, and it certainly hasn’t established the root cause.
vmanomaly UI
The anomaly might correspond to an application failure, increased traffic, a deployment, a scheduled workload, a problem with data collection, or even a perfectly legitimate but uncommon event. Statistical deviation and operational impact are not the same thing.
The analyzed time series contains spikes, stable periods, and change points. The model estimates the expected behavior of the series and can identify observations outside that expected range. However, those predictions don’t contain the business or application context necessary to explain why the deviation occurred.
That context has to come from somewhere else.
This is why I think anomaly detection becomes considerably more interesting when it is connected to the wider observability stack. The anomaly is not the answer. It is the beginning of the investigation.
From detecting anomalies to investigating them
Once unusual behavior has been identified, an AI agent can use the capabilities exposed through MCP to collect more evidence.
For example, if anomaly detection identifies an unusual period in an application’s error behavior, the agent could retrieve the corresponding metrics and determine which service is affected. It could then inspect related logs or traces for the same time period and use that information to provide additional context around the anomaly.
The important part is that the agent isn’t simply being asked, “Why did my application fail?” and expected to generate a plausible answer.
Instead, it is being asked to interact with real systems, retrieve evidence, correlate that evidence, and present its findings to the engineer.
That is a much more useful way to think about AI in observability. The model should help coordinate an investigation, rather than substitute speculation for an investigation.
A concrete example with vmanomaly
In my demo, I used vmanomaly, the anomaly detection component from VictoriaMetrics, together with an MCP server.
The example starts with observability data represented as a time series. The series can contain characteristics such as trends, recurring seasonal patterns, spikes, and change points. The anomaly detection model estimates expected behavior and highlights observations that deviate from that expectation.
Understanding these characteristics matters because not all time series behave in the same way. A metric with a strong daily pattern should not necessarily be treated in the same way as a mostly flat metric with occasional spikes. Likewise, a sustained change in baseline may tell us something very different from a single extreme data point.
The anomaly detection system gives us a way to identify these unusual behaviors, but the next question is the one that matters operationally: What does this anomaly mean for my application?
This is where connecting anomaly detection to the rest of the observability environment becomes useful.
Using MCP to extend the investigation
The MCP server exposes capabilities from the anomaly detection system to a compatible AI client. Instead of the AI assistant relying only on its existing knowledge, it can retrieve information from the running system.
The same principle can extend to other parts of the observability stack. An agent could use one tool to retrieve metrics, another to search logs, another to investigate traces, and another to access documentation or anomaly detection information.
The interesting part isn’t simply that these tools can be called using natural language. It is that the agent can potentially coordinate them as part of the same investigation.
For example, an engineer could begin with a question about unusual application behavior. The agent could retrieve the relevant anomaly information, inspect the associated telemetry, and use other available signals to provide additional context.
Anomaly Assistant with MCP in vmanomaly UI
Natural language therefore becomes an entry point into the observability stack , rather than a replacement for the technologies underneath it.
Natural language doesn’t replace query languages
This distinction is easy to lose when we talk about “chatting with observability data.”
If I ask an agent whether application errors increased during an anomalous period, the underlying observability system still needs to answer that question. Somewhere underneath the conversation, a metrics or logs query may still need to execute.
Likewise, if I ask for slow traces belonging to a particular service, the tracing backend still has to perform the search.
PromQL, LogsQL, trace queries, APIs, anomaly detection models, and the underlying databases continue doing the technical work. The AI agent does not replace those components.
What changes is the interface and potentially the coordination between them. Instead of manually navigating each system and constructing every query, the engineer can describe the investigation while the agent invokes the appropriate capabilities.
That can lower some of the friction involved in troubleshooting without hiding where the evidence actually comes from.
Keep the observability system as the source of truth
This leads to one of the most important architectural principles for this approach: the LLM should not become your observability database.
Your metrics system remains responsible for metrics. Your logging system remains responsible for logs. Your tracing backend remains responsible for traces. Your anomaly detection system remains responsible for identifying deviations in the data.
The AI layer sits above these systems and helps coordinate access to them.
This separation is useful because it keeps the observability stack as the source of truth. The agent can reason over the information returned by those systems, but it doesn’t need to reproduce or replace their functionality.
It also gives us a better way to think about hallucination in an operational context. If a question can be answered by querying the actual system, we should prefer retrieving that evidence over asking the model to produce an answer from its internal knowledge.
For production troubleshooting, retrieving evidence is more valuable than generating a plausible answer.
Security still matters
Connecting an AI agent to production observability infrastructure should be treated like connecting any other operational client.
MCP does not remove the need for authentication or authorization. In fact, giving an agent the ability to invoke tools makes careful access control even more important.
An agent should only have access to the capabilities required for its task. Read-only access should be preferred when modifications aren’t necessary, credentials should be scoped appropriately, and MCP servers should be treated as part of the operational attack surface rather than invisible AI infrastructure.
The same applies to observability of the agent’s actions. If an AI system is querying production infrastructure through MCP, those calls should be visible and auditable just like other automated interactions with production systems.
This is not a fundamentally new security problem created by AI. It is the same least-privilege principle we already apply to APIs, service accounts, CI/CD systems, and automation. What changes is that the client deciding which operation to perform may now be an AI agent.
MCP servers need observability too
There is a slightly recursive consequence of introducing MCP into an observability stack: the MCP layer itself becomes infrastructure that needs to be observed.
If an AI-assisted investigation depends on an MCP server, we need to know whether that server is available, whether requests are succeeding, whether calls to upstream systems are failing, and whether the agent is encountering timeouts or authorization errors.
This matters even more as MCP becomes part of incident investigation. A failed MCP request should not be interpreted by an agent as evidence that no relevant telemetry exists. The difference between “the query returned nothing” and “the system responsible for executing the query is unavailable” is operationally significant.
MCP servers therefore need the same basic treatment as other components in the observability path: health monitoring, appropriate telemetry, authentication, failure handling, and clear operational boundaries.
There is something slightly amusing about needing to observe the infrastructure we introduced to help AI interact with our observability infrastructure, but it is also exactly what we should expect. Once something becomes part of the production path, it needs to be operated like production infrastructure.
Keep engineers in the loop
There is a tempting story around AI operations where an agent detects a problem, diagnoses it, and then fixes production automatically.
I don’t think that needs to be our starting point.
A much more practical target is reducing the mechanical work involved in an investigation. An agent can gather telemetry, execute known queries, correlate signals, retrieve relevant documentation, follow established troubleshooting workflows, and summarize the evidence it finds.
The engineer can then decide what that evidence means and what action is appropriate.
This distinction matters because operational judgment involves more than finding correlations in telemetry. Engineers understand business context, risk, previous incidents, architectural constraints, and the consequences of making a change.
The goal isn’t to remove that judgment. It is to spend less of it on navigating between tools and repeatedly reconstructing context.
From dashboards to investigations
Dashboards aren’t going away. Neither are query languages, alerts, runbooks, or traditional observability tools.
What MCP introduces is another way for software to interact with those systems. An AI agent can potentially select and combine available capabilities according to the problem it is trying to investigate.
Anomaly detection gives that investigation a useful starting point by identifying behavior that differs from what is expected. MCP gives the agent controlled access to the live systems containing the evidence. Skills provide reusable operational knowledge about how that evidence should be investigated. The observability stack remains responsible for producing and querying the actual telemetry.
The engineer remains responsible for interpreting the result and deciding what to do next.
That combination is the part of AI-assisted observability I find most useful. It is less about asking an LLM to somehow know more about our production systems and more about giving an agent controlled access to the evidence, tools, and workflows that engineers already trust.
That is a much more practical foundation for bringing AI into observability.
Resources
→ Anomaly detection playground: https://docs.victoriametrics.com/anomaly-detection/ui/#playgrounds
→ VictoriaMetrics skills: https://github.com/VictoriaMetrics/skills
→ Vmanomaly MCP server: https://github.com/VictoriaMetrics/mcp-vmanomaly
→ Blog articles: https://victoriametrics.com/blog/observabilitys-sixth-sense-grounding-anomaly-detection-in-reality/
You can find me on:
Bluesky: @didiviking.bsky.social
X: @dianavtodea
Mastodon: @dianatodea
Instagram: @didivikingdevrel
Top comments (0)