Machine Learning Log Anomaly Detection with Grafana Loki

Originally published on kuryzhev.cloud

---

The scenario

We built machine learning log anomaly detection into our Loki stack after a credential-stuffing campaign sat undetected for four days. Our "5 failed logins from one IP in 5 minutes" alert never fired once. The attacker knew exactly how to avoid it — they rotated across roughly 3,000 residential proxy IPs, hitting our /login endpoint at maybe two requests per minute per IP. No single source ever tripped a threshold. The only sign was a slow, steady climb in aggregate 401 responses that a human eventually noticed while poking around a dashboard, purely by accident.

Our stack at the time was Grafana Loki 2.9.4 as the log store, structured JSON app logs shipped from our services, and Grafana 10.4 for visualization. Alertmanager was already wired up for Loki ruler alerts. What we didn't have was anything that looked at the shape of traffic over time — no baseline, no drift detection, nothing that would catch "this looks statistically weird" versus "this crossed a fixed line." That gap is exactly what signature-based rules can't cover, and it's why we ended up building a small anomaly-scoring sidecar rather than adding yet another static threshold.

The goal was narrow and deliberately unglamorous: score three LogQL-derived metrics — 401 rate, 5xx rate, and unique-IP churn — against a learned baseline, and fire an alert when the pattern deviates. Nothing fancy, no deep learning, no vendor SIEM subscription. Just an IsolationForest sitting on top of metrics Loki was already computing.

Prerequisites

Before touching any config, get the log shape right, because this whole approach falls apart if your ingestion is already a mess. You'll need:

• Loki 2.9.4+ with the ruler enabled and structured metadata support — this matters for keeping label cardinality sane.
• Grafana Alloy 1.4.0 shipping structured JSON logs (Promtail 2.9.x is fine if you haven't migrated yet — no need to rush that just for this).
• Python 3.11 with scikit-learn==1.4.2, pandas==2.2.1, requests==2.31.0 for the scoring service.
• Alertmanager already receiving Loki ruler alerts, plus a webhook receiver you control for the ML-generated ones.
• App logs that include status, client_ip, path, and user_id as fields inside the JSON body — not as Loki labels. This one detail decides whether the rest of this works or quietly bankrupts your Loki bill.

If your logging setup differs a lot from this, check the Loki documentation on structured metadata before proceeding — the ingestion model changed meaningfully between versions.

Step 1: Shape logs so LogQL can extract features without exploding cardinality

This is the step most teams get wrong first, and it's the one that costs real money. Keep Loki labels to low-cardinality dimensions only — app, env, level. Everything else — client_ip, user_id, status_code — goes into structured metadata or the JSON body, parsed at query time with | json.

I've seen teams label by client_ip or user_id because it makes ad-hoc querying feel easier. Don't. Each unique label combination creates a new stream in Loki, and with even moderate traffic you'll 3-5x your ingestion cost and storage on the exact same log volume. It looks harmless in a dev environment with ten users. It is not harmless in production.

Sanity-check field parsing before writing anything else:

logcli query '{app="api"} | json | status="401"' --limit=5

If that returns clean rows with a real status field, you're good. If it returns nothing or garbled JSON, fix the log shipper config first — everything downstream depends on this parsing correctly.

Step 2: Turn raw logs into time-series features via Loki's ruler

Querying raw logs over long ranges is where things fall apart. We tried running the feature queries directly against /loki/api/v1/query_range over a 30-day window and hit context deadline exceeded almost every time. Loki's ruler solves this by precomputing recording rules, so the ML service reads clean numeric time series instead of re-scanning raw logs on every cycle.

Here's the ruler config we run under /etc/loki/rules/prod/api-security.yaml:

# /etc/loki/rules/prod/api-security.yaml
# Loki ruler recording rules - precompute metrics so the ML service
# never hits raw log ranges directly (avoids query timeouts + cost).

groups:
  - name: api-security
    interval: 1m
    rules:
      - record: api_security:rate_401_5m
        expr: |
          sum(rate({app="api", env="prod"} | json | status="401" [5m]))

      - record: api_security:rate_5xx_5m
        expr: |
          sum(rate({app="api", env="prod"} | json | status=~"5.." [5m]))

      - record: api_security:distinct_ip_5m
        expr: |
          count(
            count by (client_ip) (
              rate({app="api", env="prod"} | json | __error__="" [5m])
            )
          )

# Expected verification output from Loki query API during a simulated attack:
# {
#   "resultType": "matrix",
#   "result": [{
#     "metric": {"__name__": "api_security:rate_401_5m"},
#     "values": [
#       [1716400000, "0.4"],
#       [1716400060, "0.6"],
#       [1716400120, "5.8"],   <- spike from simulated credential stuffing
#       [1716400180, "6.1"]
#     ]
#   }]
# }

These get remote-written into your metrics backend (or scraped straight off the ruler's metrics endpoint), so the scoring service downstream reads numeric series, not LogQL output.

Step 3: Build the anomaly scoring service

This is the actual ML piece, and it's deliberately boring — an IsolationForest, not a neural net. We fit it on "known good" traffic and score new windows against that baseline every 5 minutes via a systemd timer.

The biggest mistake we made on the first pass: we trained on a window that already included the ongoing attack. The model happily learned the slow credential-stuffing rate as "normal" and stopped flagging it entirely — classic data leakage. Now we explicitly exclude any tagged incident window from training data.

# anomaly_detector.py - pulls recorded rate metrics from Loki's ruler output
# and scores them for anomalies every run (invoked via systemd timer / cron)

import requests
import pandas as pd
from sklearn.ensemble import IsolationForest
from datetime import datetime, timedelta

LOKI_URL = "http://loki:3100/loki/api/v1/query_range"
LOKI_TOKEN = "REPLACE_WITH_READONLY_TOKEN"  # scoped read-only, never write access
ALERTMANAGER_URL = "http://alertmanager:9093/api/v2/alerts"

# Pull the last 7 days of the precomputed 401-rate recording rule metric
def fetch_metric(query: str, days: int = 7) -> pd.DataFrame:
    end = datetime.utcnow()
    start = end - timedelta(days=days)
    params = {
        "query": query,
        "start": int(start.timestamp()),
        "end": int(end.timestamp()),
        "step": "60s",  # 1-minute resolution matches the recording rule interval
    }
    headers = {"Authorization": f"Bearer {LOKI_TOKEN}"}
    resp = requests.get(LOKI_URL, params=params, headers=headers, timeout=30)
    resp.raise_for_status()
    result = resp.json()["data"]["result"]
    if not result:
        return pd.DataFrame(columns=["ts", "value"])
    values = result[0]["values"]
    df = pd.DataFrame(values, columns=["ts", "value"])
    df["value"] = df["value"].astype(float)
    return df.fillna(0.0)  # avoid NaNs crashing IsolationForest on sparse windows

def train_and_score(df: pd.DataFrame):
    # Exclude the most recent 30 minutes from training to avoid scoring against itself
    train_df = df.iloc[:-30]
    live_df = df.iloc[-30:]

    model = IsolationForest(contamination=0.02, n_estimators=200, random_state=42)
    model.fit(train_df[["value"]])

    live_df = live_df.copy()
    live_df["score"] = model.decision_function(live_df[["value"]])
    live_df["is_anomaly"] = model.predict(live_df[["value"]]) == -1
    return live_df

def push_alert(row):
    payload = [{
        "labels": {
            "alertname": "LogAnomalyDetected",
            "severity": "warning" if row["score"] > -0.15 else "critical",
            "app": "api",
        },
        "annotations": {
            "summary": f"401-rate anomaly, score={row['score']:.3f}, value={row['value']}",
        },
        "startsAt": datetime.utcnow().isoformat() + "Z",
    }]
    requests.post(ALERTMANAGER_URL, json=payload, timeout=10)

if __name__ == "__main__":
    metric_query = 'api_security:rate_401_5m'  # name of the ruler-recorded series
    df = fetch_metric(metric_query)
    scored = train_and_score(df)
    for _, row in scored[scored["is_anomaly"]].iterrows():
        push_alert(row)

Watch out for sparse overnight windows — near-zero traffic buckets can produce NaNs that crash score_samples outright. The fillna(0.0) above is doing real work, not decoration.

Step 4: Wire alerts back into Alertmanager and Grafana

A model that just logs "anomaly detected" to a file nobody reads is worthless. The scoring service POSTs directly to Alertmanager's /api/v2/alerts endpoint with labels alertname=LogAnomalyDetected, severity, and app, so it routes through the exact same on-call pipeline as every other alert. We also annotate the Grafana 10.4 dashboard panel via the annotations API so anomaly points show up directly on the raw rate graph — that visual overlay makes triage much faster than cross-referencing timestamps by hand.

One thing I'll flag hard: the scoring service only needs a read-only Loki token, never write access. And the Alertmanager webhook needs real authentication — shared secret or mTLS. An open webhook here means anyone who finds the URL can inject fake criticals and burn out your on-call rotation with alert fatigue. We learned this the hard way on an internal pentest exercise where a teammate spoofed three "critical" pages just to prove the point.

Verify and test

Don't trust this pipeline until you've watched it catch something on purpose. We wrote a small script that hits /login from a rotating pool of IPs at low RPS to simulate the exact slow credential-stuffing pattern that started this whole project. Confirm the recorded rule shows a rising 401-rate in Grafana Explore first — if the recording rule itself isn't moving, the ML layer never gets a chance.

Next, check the scoring service logs for a score crossing the decision threshold, and confirm the alert actually landed in Alertmanager:

amtool alert query alertname=LogAnomalyDetected

Then run a full quiet baseline day with zero simulated traffic and confirm zero false positives. If noise shows up, tune contamination down from 0.02. If a genuinely slow attack slips through undetected, raise it slightly and re-test. And re-check your baseline after any deploy that meaningfully changes normal traffic shape — a new feature release, a marketing spike, a mobile app update — because those will all look like anomalies to a model trained on last week's patterns.

Treat this whole setup as a complement to signature-based detection, not a replacement for it — the fixed-threshold rules still catch the loud, obvious stuff instantly, while machine learning log anomaly detection on Loki catches the slow-burn attacks that never trip a single line. We retrain nightly on a rolling 7-day window, explicitly excluding any tagged incident windows, and we watch for baseline drift after every deploy that touches user-facing traffic. The recording-rule layer is what makes this affordable at any real scale — without it you're paying to re-scan raw logs on every scoring cycle, which is a good way to turn a clever detection idea into a Loki bill nobody wants to explain. If you're weighing this against other log-analysis approaches for your stack, it's worth browsing more DevOps setup notes on kuryzhev.cloud before committing to an architecture.

Related

• More on building alerting pipelines and dashboard tuning
• Security hardening patterns for public-facing services
• Python automation scripts for DevOps tooling