Originally published on kuryzhev.cloud
When You Face This Choice
AI log anomaly detection becomes a real conversation the moment your threshold-based alerts fail you at the worst possible time. Last quarter our Prometheus/Grafana stack was screaming about CPU and disk, but a slow memory leak in a checkout service crept past every static rule for six hours before it finally OOM-killed the pod at 3am. Nobody paged. Static thresholds don't catch gradual drift — they catch cliffs.
Once you decide to bring "AI" into log analysis, you hit a fork: train your own statistical ML model on log-derived metrics (Isolation Forest, Prophet, Elastic ML), or pipe logs and log summaries into an LLM for semantic reasoning. Both get marketed as "AI-powered anomaly detection," but they solve different problems at wildly different costs.
The variable that actually forces the decision isn't taste — it's constraints. How many GB/day of logs are you generating? Do you need sub-second paging or is this post-incident triage? And critically: do your stack traces contain API keys, session tokens, or PII that you can't legally ship to a third-party API? Answer those three questions honestly before you pick a tool, because retrofitting compliance after you've already piped six months of logs to GPT-4o is a much worse Tuesday.
Option A — Statistical/ML Anomaly Detection (Isolation Forest, Prophet, Elastic ML)
This is the boring, battle-tested path, and I mean that as a compliment. You extract numeric features from logs — error_count, p95_duration, unique_error_types per one-minute window — and let an unsupervised model like scikit-learn's IsolationForest score them. PyOD wraps a dozen of these algorithms if you want more than Isolation Forest. Prophet handles univariate trend/seasonality forecasting well if you're watching a single metric like request rate. Elastic ML jobs do the same thing natively inside Kibana if you're already on the Elastic stack.
Pros: scoring is sub-second, runs comfortably on a $20/month CPU box, every anomaly is explainable via feature importance ("error_count spiked to 47, three standard deviations above baseline"), and nothing leaves your network. That last point matters more than people admit until a security review asks about it.
Cons: you're doing manual feature engineering — parsing log levels, encoding severity, building rolling windows — and it's brittle. Change your log schema during a deploy and your feature pipeline silently breaks. Prophet in particular only handles univariate series; feed it correlated multi-service metrics and it'll happily generate a confident, wrong forecast with no error thrown.
Gotcha: these models drift. Retrain weekly at minimum — after every deploy cycle your baseline traffic pattern shifts, and contamination='auto' in IsolationForest is way too conservative for log data. We tune it manually to 0.01–0.05 and still revisit it monthly. Skip retraining and your false-positive rate climbs until on-call mutes the channel — which defeats the entire point.
Option B — LLM-Based Log Analysis (GPT-4o / Claude / self-hosted Llama)
This is the seductive option. No feature engineering — you dump a chunk of logs at GPT-4o and ask "is anything wrong here?" It genuinely understands semantic context in a way statistical models never will: it'll tell you "this stack trace pattern looks like DB connection pool exhaustion, not a network timeout" without you writing a single regex. It can also generate a readable incident summary for the postmortem doc, which saves real human time.
Pros: near-zero setup, strong root-cause narrative quality, great for post-incident triage where a human is going to read the output anyway.
Cons: the economics fall apart fast at volume. GPT-4o-mini runs roughly $0.15 per 1M input tokens. Push 50GB/day of raw logs through that and you're looking at $300–600/month in tokens alone — before you've caught a single real incident. Latency is another wall: typical response time is 2–5 seconds per request, which is unusable if you need sub-second paging on a streaming pipeline. And root-cause attribution can hallucinate — it'll confidently name the wrong service.
Gotcha: logs are full of secrets. API keys, session tokens, and PII show up in stack traces constantly. Sending raw logs to a third-party LLM API without a redaction step (regex-based, or something like Presidio) is a compliance incident waiting to happen. I've seen a staging log leak an AWS secret key into a support ticket transcript this way — don't let it become production.
Decision Matrix
Here's how the two options actually stack up once you price in volume, not vibes.
Criteria | Isolation Forest/Prophet | LLM (GPT-4o/Claude)
-----------------------|--------------------------|----------------------
Setup time | 1-2 days | 1-2 hours
Cost @ 50GB/day | ~$20-50/mo (CPU box) | $300-600/mo (tokens)
Latency | <1s | 2-5s
Explainability (code) | Feature importances | Natural language
Explainability (human) | Weak | Strong
False-positive tuning | Manual, but predictable | Prompt-dependent
Data residency | Stays in-network | Third-party API risk
Maintenance | Weekly retraining | Prompt/version drift
The pattern is obvious: Option A wins on cost, latency, and security every time. Option B wins on setup speed and how good the output reads to a human who wasn't paged at 3am. Mini decision tree: high-volume, real-time infra metrics that need to page someone → Isolation Forest/Prophet. Low-volume, post-incident root-cause analysis where a human reads the summary anyway → LLM.
My Pick
I don't do "it depends" here — I run a hybrid pipeline, and I think pure-LLM log analysis at production scale is a cost and latency trap you should actively avoid for alerting. Isolation Forest does the cheap, real-time scoring on feature vectors and flags anomalous windows. Only those flagged windows — not the raw log firehose — get forwarded to an LLM for human-readable triage. That's the difference between $600/month in tokens and maybe $15.
The feature engineering script below builds one-minute windows from structured logs and flags anomalies with a manually tuned contamination rate:
# feature_engineering_and_detect.py
# Builds per-minute feature vectors from structured logs, then runs
# Isolation Forest to flag anomalous windows. Run on a cron every 5 min.
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
import json
import sys
# Load parsed logs (assumes Fluent Bit already shipped JSON logs to a file/ES)
# Each record: {"ts": "...", "level": "ERROR", "duration_ms": 240, "service": "checkout"}
def load_logs(path):
records = []
with open(path) as f:
for line in f:
try:
records.append(json.loads(line))
except json.JSONDecodeError:
continue # skip malformed lines, don't crash the whole batch
return pd.DataFrame(records)
def build_features(df):
df["ts"] = pd.to_datetime(df["ts"])
df["window"] = df["ts"].dt.floor("1min")
grouped = df.groupby("window").agg(
error_count=("level", lambda x: (x == "ERROR").sum()),
unique_errors=("level", lambda x: x[x == "ERROR"].nunique()),
p95_duration=("duration_ms", lambda x: x.quantile(0.95)),
request_count=("ts", "count"),
).reset_index()
# gotcha: NaNs from empty windows will break IsolationForest.fit()
grouped = grouped.fillna(0)
return grouped
def detect_anomalies(features):
feature_cols = ["error_count", "unique_errors", "p95_duration", "request_count"]
scaler = StandardScaler()
X = scaler.fit_transform(features[feature_cols])
# contamination tuned manually — 'auto' is too conservative for log data
model = IsolationForest(n_estimators=200, contamination=0.03, random_state=42)
features["anomaly_score"] = model.fit_predict(X) # -1 = anomaly, 1 = normal
return features[features["anomaly_score"] == -1]
if __name__ == "__main__":
df = load_logs(sys.argv[1])
features = build_features(df)
anomalies = detect_anomalies(features)
print(anomalies[["window", "error_count", "p95_duration"]].to_json(orient="records"))
And here's the Fluent Bit config that only routes flagged, filtered batches — never the raw firehose — to the LLM triage endpoint:
# fluent-bit.conf snippet: route flagged anomaly windows to an LLM
# triage webhook instead of shipping raw logs to any API.
[INPUT]
Name tail
Path /var/log/app/*.json
Tag app.logs
Mem_Buf_Limit 50MB # too low here = silently dropped logs on spikes
[FILTER]
Name grep
Match app.logs
Regex level ERROR|WARN
[OUTPUT]
Name http
Match app.logs
Host anomaly-triage.internal
Port 8080
URI /triage
Format json
# only anomaly-flagged batches hit this endpoint, which then
# calls the LLM API — raw firehose never leaves the cluster
# --- Example anomaly_score output from Isolation Forest run ---
# {"window": "2024-06-01T03:14:00", "error_count": 47, "p95_duration": 3200}
# {"window": "2024-06-01T03:15:00", "error_count": 52, "p95_duration": 4100}
# Only these two windows get forwarded to GPT-4o for human-readable triage,
# not the full 50GB/day log stream.
If compliance blocks external APIs entirely — which happens more often than vendors admit — self-hosted Llama 3 8B via Ollama is the fallback. Accept that root-cause reasoning quality drops noticeably versus GPT-4o; you're trading accuracy for data residency, and that's usually the right trade. If you need better quality self-hosted, vLLM serving Llama 3 70B works, but budget for A100/H100-class GPUs — it's not a side project anymore. We cover more of our own alerting stack decisions over on kuryzhev.cloud if you want the full context on how this fits into our on-call setup. Bottom line: let cheap ML do the watching, let the LLM do the explaining, and never let raw logs be the thing you're paying per-token for.
Top comments (0)