Fix Predictive Alerting False Positives in Prometheus Python Models

#python #monitoring #devops

Originally published on kuryzhev.cloud

Your anomaly model tested perfectly offline — 94% precision on a historical CSV export, clean ROC curve, the works. Then it went live wired to real query_range calls, and within a day it was paging on-call for pod restarts and empty scrape windows. This is one of the most common failure modes we've hit rolling out predictive alerting on top of Prometheus, and it almost never shows up in the training notebook. It shows up in production, at 3am, as noise.

Symptoms

The tell is usually a mismatch between what the model was trained to expect and what it actually receives at inference time. On our setup (Prometheus 2.48.x, Python 3.11, pandas 2.1.x, scikit-learn 1.4.0) we saw four recurring patterns:

Predictive alerts firing right after a Prometheus restart or a scrape gap — the model scores a stale window as a spike.
Alertmanager receiving duplicate or late webhook calls from the scoring service, or worse, the service timing out and the notification just disappearing.
Offline accuracy that looked great degrading into pure noise once wired to live data — predictions flatline into a constant class or oscillate randomly.
CPU/memory spikes on the scoring pod that correlate with label churn (a rolling deploy changing pod_name values), not with actual traffic load.

If any of these sound familiar, don't tune the model threshold first. The threshold isn't the problem — the input pipeline is. We burned two days adjusting predict_proba cutoffs before realizing the features themselves were garbage.

Root cause

Every case above traces back to the same thing: the query that built the training set doesn't behave the same way as the query that feeds live inference. A few specific mechanisms:

Step/resolution drift. Training pulled data at one step, inference queries at another. Different aggregation windows, different NaN alignment, different feature values for the "same" metric.
Counter vs rate confusion. Feeding raw _total counters straight into the model instead of rate()/increase(). A pod restart resets the counter to zero — the model sees a massive drop that looks exactly like an anomaly, because to a naive feature vector, it is one.
Staleness handling. Prometheus marks a series stale after ~5 minutes without a scrape. The client returns NaN, and if you don't handle that explicitly, NaNs either crash predict() or silently poison the row.
Cardinality explosion. An open label like pod_name or request_id can turn a 50-series query into 5,000+ series after a rolling deploy. The scorer wasn't trained for that shape and either errors out or, worse, quietly reshapes and feeds nonsense into the model.

We also got bitten by the prometheus-api-client (pip, v0.5.3) MetricRangeDataFrame helper — it silently drops metrics with mismatched label sets. No exception, no warning. It just returns fewer columns than you expect, and the model happily scores whatever it gets.

Fix #1 — Align query resolution, step, and rate semantics

Make the live query mathematically equivalent to the one that trained the model. This is non-negotiable and should be treated as a contract, not a tunable.

Pin step to the exact resampling interval used in training (e.g. 5m, not 15s). Document it as a named constant, not a magic string scattered across scripts.
Never post-process raw counters in pandas after the fact — wrap them in rate(metric[5m]) at query time, in PromQL, before they ever reach Python.
Interpolate short gaps with ffill(limit=2), but hard-fail on longer gaps. Don't zero-fill and don't alert blind — skipping a scoring cycle is safer than scoring garbage.

Gotcha: rate() needs at least two samples inside the range. If your scrape interval and range window are both 5m, you'll get NaN back, not an error. Always keep the range at least 2x the scrape interval.

Here's the scorer with these constraints baked in:

#!/usr/bin/env python3
# scorer.py — pulls aligned Prometheus features and runs predictive alerting inference
import hashlib
import sys
import time
from datetime import datetime, timedelta

import joblib
import pandas as pd
import requests

PROM_URL = "https://prometheus.internal:9090"
STEP = "5m"              # MUST match training resampling interval — do not change casually
LOOKBACK = timedelta(hours=2)
FEATURE_QUERIES = {
    # use recording rules, not raw metrics, to avoid schema drift
    "cpu_rate": 'job:cpu_usage:rate5m{namespace="prod"}',
    "err_rate": 'job:http_errors:rate5m{namespace="prod"}',
    "req_rate": 'job:http_requests:rate5m{namespace="prod"}',
}
EXPECTED_FEATURE_HASH = "9a1c7f...trunc"  # computed once at training time

def fetch_range(promql: str, start, end):
    resp = requests.get(
        f"{PROM_URL}/api/v1/query_range",
        params={"query": promql, "start": start.timestamp(), "end": end.timestamp(), "step": STEP},
        timeout=10,  # fail fast — don't hang the whole pipeline on one bad query
        verify="/etc/ssl/certs/prom-ca.pem",
    )
    resp.raise_for_status()
    result = resp.json()["data"]["result"]
    if not result:
        raise ValueError(f"empty result for query: {promql}")
    if len(result) > 1:
        raise ValueError(f"cardinality guard tripped: {len(result)} series for {promql}")
    return pd.Series(
        {pd.to_datetime(ts, unit="s"): float(val) if val != "NaN" else float("nan")
         for ts, val in result[0]["values"]}
    )

def build_feature_frame():
    end = datetime.utcnow()
    start = end - LOOKBACK
    series = {name: fetch_range(q, start, end) for name, q in FEATURE_QUERIES.items()}
    df = pd.DataFrame(series)
    df = df.ffill(limit=2)          # tolerate short scrape gaps only
    if df.isna().any().any():
        raise RuntimeError("unrecoverable data gap — skipping inference, not alerting blind")
    return df

def verify_feature_contract(df):
    computed = hashlib.sha256(",".join(sorted(df.columns)).encode()).hexdigest()[:8]
    if computed != EXPECTED_FEATURE_HASH[:8]:
        raise RuntimeError(f"feature schema drift: {computed} != {EXPECTED_FEATURE_HASH[:8]}")

if __name__ == "__main__":
    df = build_feature_frame()
    verify_feature_contract(df)
    model = joblib.load("model_v3_feat9a1c_skl1.4.0.pkl")
    score = model.predict_proba(df.tail(1))[0][1]
    if score > 0.85:
        requests.post("http://alertmanager:9093/api/v2/alerts", json=[{
            "labels": {"alertname": "PredictiveAnomaly", "severity": "warning"},
            "annotations": {"score": str(round(score, 3))},
        }], timeout=5)  # keep well under Alertmanager's retry window

Fix #2 — Freeze the feature contract and detect schema drift

This is the fix that saved us the most future pain. Once you've aligned queries, someone will still eventually rename a metric or edit a relabel config, and the model input will silently change shape or meaning. You need to catch that before it reaches predict(), not after the alerts stop making sense.

Move feature extraction behind stable Prometheus recording rules using a level:metric:operation naming convention (e.g. job:http_requests:rate5m). Raw metric renames and relabeling then can't leak into the model input without a deliberate rule change.
Hash the ordered feature-name list at training time with hashlib.sha256. On every inference run, recompute the hash and refuse to score if it diverges. This is cheap — a few microseconds — and it catches silent renames immediately.
Version the model artifact filename with the feature hash and the sklearn version, e.g. model_v3_feat9a1c_skl1.4.0.pkl. sklearn 1.4.0 loading a pickle saved under 1.3.x throws InconsistentVersionWarning, and if you're not watching logs closely that warning gets ignored until predictions quietly drift.

Recording rules and the failure output when the hash check trips:

# recording_rules.yml — stabilizes feature inputs against relabeling/rename churn
groups:
  - name: ml-feature-inputs
    interval: 5m           # MUST equal STEP in scorer.py
    rules:
      - record: job:http_errors:rate5m
        expr: sum by (namespace) (rate(http_requests_total{status=~"5.."}[5m]))
      - record: job:http_requests:rate5m
        expr: sum by (namespace) (rate(http_requests_total[5m]))
      - record: job:cpu_usage:rate5m
        expr: sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

# --- Example failure output when schema drift check trips ---
# $ python scorer.py
# Traceback (most recent call last):
#   ...
# RuntimeError: feature schema drift: 3fbb21a4 != 9a1c7f00
# (cause: someone renamed job:cpu_usage:rate5m -> job:cpu:rate5m in a rules PR)

Watch out: a joblib-saved model loaded in a container with a different libopenblas/numpy build under the hood can produce silently different float precision — same code, subtly different scores. Pin the base image by digest, not just by tag, if your scoring depends on tight probability thresholds.

Fix #3 — Bound cardinality and guard the query itself

The third failure mode is operational, not statistical: an unbounded query can crash the scorer or hammer the Prometheus server before the model ever gets involved.

Use explicit label matchers ({namespace="prod", pod=~"api-.*"}) instead of open wildcards. Reject and log any query returning more series than the model's trained feature width — don't try to reshape and guess.
Respect Prometheus's --query.timeout (default 2m) and set a client-side timeout that's comfortably under half of Alertmanager's webhook retry window. If the scorer hangs, Alertmanager retries the webhook and you get duplicate alerts, or the notification just gets dropped silently.
Downsample long lookback windows via Thanos (0.34 in our stack) or recording rules instead of pulling raw 15s-resolution data. Pulling 30 days at 15s across 500 series is roughly 86 million points in a single query_range call — that's not a training pipeline, that's a denial-of-service against your own monitoring stack.

We also lock the Prometheus HTTP API behind mTLS or an authenticated reverse proxy — see the official Prometheus HTTP API docs for what's exposed by default. An open /api/v1/query_range endpoint lets anyone with pod network access read internal metric names and values, which becomes a real exfil risk the moment external services depend on it for predictive alerting.

Prevention

Once the pipeline is stable, the job shifts to keeping it honest over time. Three things we now run as standard practice on every predictive alerting deployment:

Ship model health back into Prometheus itself: ml_scorer_input_freshness_seconds and ml_scorer_feature_hash_mismatch_total. Alert on the alerting system — if the scorer silently stops running, you want to know before "no anomalies" gets misread as "everything's fine." Alertmanager's default group_wait/repeat_interval backoff can mask exactly that kind of outage.
Run shadow/canary scoring for one to two weeks after any feature or model change. Log predictions, don't page on them, then compare against real incidents before flipping paging back on. This caught at least two bad deploys for us before they reached on-call.
Treat the Prometheus query endpoint as a production API with an SLA, not an internal debug tool. That means auth, timeouts, and rate limits — the same rigor you'd apply to a customer-facing service, documented on our DevOps_DayS monitoring notes.

Predictive alerting on Prometheus data can genuinely cut noise and catch things rule-based thresholds miss — but only if the feature pipeline is treated as strictly as the model itself. Get the query semantics, schema contract, and cardinality bounds locked down first. The model tuning is the easy part; it just doesn't feel that way until the pipeline underneath it stops lying.