Grafana Assistant Watchers Enter Public Preview: Always On AI Agents Come to Observability

Grafana Assistant Watchers are entering public preview for Grafana Cloud users, and the product is aimed at a gap traditional alerting has never handled particularly well.

Static alerts are excellent when the team already knows what failure looks like.

Set a threshold. Define a condition. Route the notification.

The harder problem is noticing behavior nobody thought to encode beforehand.

Assistant Watchers are designed to continuously look for unusual activity in telemetry and notify operators when something appears worth attention.

That sounds attractive.

It also raises a difficult question: how much uncertainty are teams willing to put into the alert path?

Static alerts are precise because humans define the condition

Traditional alerting survives because it is predictable.

A team decides that an error rate above a threshold for a defined period matters. The rule evaluates data. When the condition is met, the system notifies the right people.

This works best for known failure modes.

Disk space is low.

Latency crosses a target.

A service stops responding.

A queue grows beyond a safe level.

The weakness is that humans have to imagine the condition before the system can detect it.

Many incidents begin with a signal that was visible but did not match an existing rule. A new log pattern appears. A metric moves strangely but stays below the static threshold. A feature behaves differently in one namespace. Nothing technically violates the alert configuration, so nobody is paged.

Watchers target that space.

Grafana says users can describe what they want watched, such as a service, namespace, or feature, and the Assistant will calibrate a Watcher around the available telemetry.

The agent then runs on a schedule and looks for unusual behavior.

That is a different contract from alerting.

Anomaly detection is useful only when noise stays under control

The obvious danger is alert fatigue.

“Unusual” is not the same thing as “important.”

Production systems behave differently during deployments, traffic peaks, customer events, maintenance, retries, and countless other normal conditions. A system that reports every anomaly quickly becomes background noise.

Grafana appears aware of that problem.

The announcement says a Watcher can be recalibrated conversationally through Grafana Assistant or the Grafana Slack bot. It also keeps short term memory of ongoing issues so the user does not receive repeated notifications while the same problem continues.

That memory feature is more important than it may sound.

Traditional alert systems often need explicit grouping, inhibition, deduplication, and recovery logic to avoid repeated pages. An agent that understands an ongoing issue as one evolving event could reduce duplicate notifications in a more flexible way.

But the behavior will need real testing.

Short term memory creates its own questions.

How long does the system consider two anomalies part of the same issue?

When should it notify again?

Can a slowly worsening condition be hidden because the Watcher believes it already told the team?

Public preview is exactly the stage where those edges should be discovered.

Watchers can connect detection directly to investigation

The most interesting integration is with Assistant Investigations.

If Investigations are enabled, a Watcher that finds something unusual can automatically start an investigation.

That creates a more complete agentic observability loop.

First, detect behavior that does not match expectations.

Then gather evidence across telemetry.

Form and test hypotheses.

Produce findings and recommended next steps.

For operators, that could mean receiving more than an alert.

Instead of waking up to “something is wrong with this metric,” the engineer may receive a notification alongside an already started investigation containing relevant logs, traces, metrics, and hypotheses.

That is a meaningful shift.

The first minutes of an incident are often consumed by context gathering. If the system can do useful evidence collection before the human arrives, the time to understanding could shrink.

The danger is equally obvious.

If the detection is wrong and the investigation confidently builds on the wrong premise, the automation may generate an impressive pile of irrelevant evidence.

Agentic workflows compound both usefulness and error.

Watchers should complement alerts, not replace them

Grafana itself frames alerting as foundational.

That is the right position.

Known critical conditions should remain deterministic.

If a payment service is down, a team should not wait for an AI agent to decide whether the behavior looks unusual.

If a certificate expires in three days and the organization already knows that is dangerous, a clear alert rule is better.

Watchers make more sense around uncertainty.

They can explore telemetry where a fixed rule would be too broad, too noisy, or too difficult to write. They may surface new error patterns, behavioral shifts, or interactions that were never encoded into the alert catalog.

Once a useful pattern becomes well understood, the team may even decide to turn it into a normal deterministic alert.

In that sense, Watchers could become discovery tools for future monitoring policy.

The agent finds a recurring issue.

Humans decide whether it deserves a hard rule.

Always on AI changes the observability trust model

The phrase “always on cloud agents” is more important than the feature list.

A chatbot waits for a question.

A Watcher acts without one.

That changes the risk.

The system is now spending resources, interpreting behavior, deciding when something is notable, and initiating notifications continuously.

Teams will need to evaluate false positives, missed issues, cost, permission boundaries, calibration effort, and whether the short term memory behaves sensibly during long incidents.

They will also need to decide what data the agent is allowed to inspect.

Public preview gives Grafana room to learn how operators react when AI moves from an on demand assistant into the monitoring loop itself.

If the product works well, it could cover one of observability's oldest blind spots: the failure nobody thought to alert on.

If it becomes noisy, teams will disable it like any other unreliable monitor.

That makes the success criterion refreshingly simple.

An always on agent does not need to sound intelligent.

It needs to notice the few things humans genuinely wish they had seen earlier.