Originally published on kuryzhev.cloud
Your "AI-powered" anomaly alerts are probably flapping every 60 seconds because nobody set a for duration on the rule. We hit this exact problem three months ago when a client asked us to add "smart alerting" to their checkout service dashboards. What they got, at first, was a Slack channel firing and resolving the same alert every evaluation cycle — Grafana's default is 1 minute. AI anomaly detection Grafana integrations get sold as magic, but the underlying pipeline is closer to a stats class than a neural network, and treating it otherwise is where teams get burned.
What this actually does
Strip away the marketing and the pipeline looks like this: metrics come in from Prometheus, Mimir, or Loki, get aggregated into a feature (rate, count, ratio), get scored by a model, and the score itself becomes a new time series that Grafana alerts on like anything else. That's it. No LLM is involved, and it doesn't need to be — this is time-series forecasting and outlier detection, not natural language generation.
There are three real tiers of implementation, and picking the wrong one for your scale is the first mistake teams make. Grafana's built-in anomaly detection transformation, available since Grafana 10.x, uses Seasonal-ESD under the hood — no external service needed, and it's fine for a handful of dashboards. The middle tier is lightweight ML: Prophet or Kats running as a sidecar, doing seasonal forecasting on a schedule. The heavy tier is a proper serving pipeline — models exported to ONNX, served via ONNX Runtime or TorchServe, queried by a custom datasource or Alertmanager webhook.
Most infra metrics don't need Prophet. It's a heavyweight forecasting library built for business time series with strong seasonality — daily active users, sales curves. Applying it to CPU or request-rate metrics with 15-second scrape intervals is like using a sledgehammer to hang a picture frame. A z-score or exponentially weighted moving average gets you 90% of the value with a fraction of the compute cost. I stopped recommending Prophet for infra alerting after watching a Kats install drag in pandas 1.x and break a container image that already had pandas 2.x pinned for a different service — version conflicts you don't need in an alerting hot path.
How people use it wrong
Mistake one: feeding raw, unaggregated metrics into a global model. Per-pod CPU during a rolling deploy or a horizontal autoscale event looks exactly like an anomaly to any outlier detector — because it is one, just not the kind you care about. Aggregate to service level first. Score the aggregate, not the noise underneath it.
Mistake two: no hysteresis. Grafana Alerting evaluates rules on every interval, default one minute, and if your alert condition is "anomaly score above 0.75, fire immediately," you'll get an alert that fires and resolves within the same minute because the model recomputes the score on every scrape and it dances around the threshold. This is the single most common complaint I've seen in postmortems for these setups — an on-call engineer gets paged, checks the dashboard, sees green, and the alert already resolved itself. That erodes trust in the whole system faster than a false negative would.
Mistake three: naive scheduled retraining. If you retrain your model every night at 2am without checking for seasonality, you'll get weeks of false positives around month-end batch jobs, Black Friday, or payroll runs — anything that's a recurring but infrequent pattern gets treated as noise the model hasn't seen yet, then flagged as anomalous every single time it recurs. We watched this happen at a fintech client where the batch settlement job triggered "critical" anomaly alerts for three consecutive month-ends before anyone connected the dots.
The correct approach
Separate concerns. Don't try to do inference inside Grafana — it's a visualization and alerting layer, not a model server. Run a dedicated scoring service instead. Here's a sidecar using River, an online machine learning library that updates incrementally without full retrains, which matters a lot for infra metrics that drift slowly over time:
# anomaly_scorer.py — lightweight sidecar that reads Prometheus,
# computes an online anomaly score, and pushes it back via remote_write
import time
import requests
from river import anomaly, preprocessing
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
PROM_QUERY_URL = "http://prometheus:9090/api/v1/query"
PUSHGATEWAY_URL = "http://pushgateway:9091"
SERVICE_LABEL = "checkout"
QUERY = 'rate(http_requests_total{service="checkout"}[1m])'
# online model: scales input then flags outliers incrementally (no batch retrain needed)
model = preprocessing.StandardScaler() | anomaly.HalfSpaceTrees(
n_trees=25, height=8, window_size=250, seed=42
)
registry = CollectorRegistry()
score_gauge = Gauge(
"anomaly_score", "Online anomaly score", ["service"], registry=registry
)
def fetch_metric():
resp = requests.get(PROM_QUERY_URL, params={"query": QUERY}, timeout=5)
resp.raise_for_status()
result = resp.json()["data"]["result"]
if not result:
return None
return float(result[0]["value"][1])
def run():
while True:
value = fetch_metric()
if value is not None:
x = {"value": value}
score = model.score_one(x) # anomaly score, higher = more anomalous
model.learn_one(x) # incremental update, no full retrain
# normalize score to 0-1 range for consistent alert thresholds
normalized = min(max(score, 0.0), 1.0)
score_gauge.labels(service=SERVICE_LABEL).set(normalized)
# push to gateway so Prometheus scrapes it like any other metric
push_to_gateway(PUSHGATEWAY_URL, job="anomaly_scorer", registry=registry)
time.sleep(15) # match scrape interval to avoid stale scores
if __name__ == "__main__":
run()
The anomaly score becomes a plain Prometheus metric — anomaly_score{service="checkout"} 0.87 — so Grafana alerts on it exactly like it would alert on request latency. Then the alert rule uses a rolling average, not the instantaneous value, and a for duration to require a sustained breach:
# provisioning/alerting/rules.yaml — Grafana-as-code alert rule
# reads the anomaly_score metric, requires sustained breach (no flapping),
# and combines with an SLO burn-rate check via Alertmanager routing
apiVersion: 1
groups:
- orgId: 1
name: infra-anomaly-alerts
folder: AI-Anomaly-Detection
interval: 1m
rules:
- uid: anomaly-checkout-score
title: "Checkout service anomaly score elevated"
condition: C
data:
- refId: A
datasourceUid: prometheus
model:
expr: avg_over_time(anomaly_score{service="checkout"}[5m])
- refId: C
datasourceUid: __expr__
model:
type: threshold
expression: A
conditions:
- evaluator:
type: gt
params: [0.75]
for: 5m # sustained breach required, prevents flapping on single spikes
labels:
severity: warning
model_version: v3
annotations:
summary: "Anomaly score {{ $values.A }} exceeded 0.75 for 5m on checkout"
runbook_url: "https://kuryzhev.cloud/runbooks/anomaly-checkout"
Watch out: version-pin your model artifacts. A model_v3.onnx file with a SHA256 checksum committed alongside the deployment manifest turns your alert behavior into something reproducible and auditable. Treat it like infrastructure code, not a throwaway notebook experiment — because when someone asks "why did this fire at 3am," you need an answer better than "the model changed."
Advanced patterns
Once you're past a single service, run per-service models with labels — {service="checkout", model_version="v3"} — so dashboards and alert rules template cleanly across teams without duplicating pipelines. Each team owns their model version, and rollbacks are just a label change.
Ensemble alerting cuts false positives dramatically. Combine the anomaly score with a static SLO burn-rate alert using Alertmanager's nested routing tree — require both to fire before paging. A rough sketch: a parent route matches severity=warning and routes to two child branches, one matching alertname=AnomalyScore and one matching alertname=SLOBurnRate, with a receiver that only triggers when both are active in the same group. This is the pattern that finally got our fintech client's false-positive rate down from daily noise to a handful of real pages a month.
Close the loop with feedback. Grafana OnCall logs acknowledge/dismiss actions on incidents — capture those as labeled ground truth and feed them into periodic supervised fine-tuning of your model. This is the difference between a static anomaly detector and one that actually learns your service's quirks over time, including the seasonal patterns that caused mistake three above.
Performance notes
Keep inference under 200ms per series. Grafana evaluates alert rules synchronously per interval, and the default evaluation timeout is 30 seconds — if your scoring datasource is slow, you'll see evaluation failed: context deadline exceeded in the Grafana logs, and that's a symptom worth taking seriously before it becomes a missed page. ONNX Runtime 1.16+ handles small forecasting or outlier models comfortably within that budget.
Cost matters more than people expect. Streaming per-service inference at 15-second resolution across 200+ services can multiply active series count 3-5x in Mimir or Cortex, since every score is a new time series with its own cardinality. Batch retraining hourly instead of continuously streaming updates is usually the right tradeoff — you lose a bit of freshness, you keep your storage bill sane.
Scope the scoring service's credentials narrowly. It needs read access to raw metrics and write access to push scores back — that's real blast radius if the token leaks. Use a dedicated remote_write tenant or namespace rather than an admin-level API key. A sidecar with broad Prometheus access and network exposure is exactly the kind of thing that becomes a lateral-movement vector during an incident, and nobody wants to explain that in a postmortem. If you're running Grafana in HA, also double-check that the anomaly_score metric is computed consistently across replicas — inconsistent scoring per instance produces alert state that looks like a bug even when the pipeline is technically working. More on wiring alerting pipelines end to end is covered in our DevOps_DayS observability posts.
Top comments (0)