Last month I ran hundreds of LangChain agent calls in production. Some of them silently failed by using wrong tool sequences, latency spikes, or even hallucinated outputs. My logs showed zero errors. No exceptions. No warnings.
The agent just did the wrong thing, quietly.
Traditional monitoring tools weren't built for this. Datadog can tell you a function threw an exception. It can't tell you your agent called delete_file when it's never done that before, or that your LLM is suddenly generating 10x more tokens than usual, or that output quality has been slowly degrading over the last 500 runs.
So I built Drift.
What Drift Does
Drift hooks into your agent's execution and applies statistical anomaly detection to the event stream in real time. Three detectors run simultaneously:
Latency & token SPC — Uses rolling z-scores to flag when a tool call or LLM response takes significantly longer or uses significantly more tokens than its baseline. Catches hung API calls, runaway generation, and upstream provider issues.
Sequence anomaly detection — Builds a Markov transition matrix of tool-call sequences and flags when the agent takes a path that's never or rarely been seen. Catches agents going off-script, skipping required steps, or making dangerous tool calls they've never made before.
Output drift detection — Tracks output length, vocabulary diversity, and structural patterns over time. Flags when outputs shift significantly from baseline. Catches hallucination drift, prompt injection effects, and gradual quality degradation.
Three Lines to Add It
bash
pip install drift-detection
python
`from drift import DriftGuard
from drift.callbacks.langchain import DriftCallbackHandler
guard = DriftGuard(on_anomaly=lambda a: print(f"🚨 {a}"))
handler = DriftCallbackHandler(guard)
Add to any LangChain agent, chain, or LLM
agent.run("your query", callbacks=[handler])
guard.report()`
What It Looks Like in Action
DRIFT DEMO — Agent Anomaly Detection
[Phase 1] Building baseline with 20 normal agent runs...
✓ 120 events processed, 0 anomalies
[Phase 2] Injecting anomalies...
--- Injecting: Latency spike on 'search_web' ---
🚨 [CRITICAL] latency_spike: 'search_web' latency is 2500.0ms,
76.3σ above mean (200.7ms ± 30.1)
--- Injecting: Novel tool sequence (search_web → delete_file) ---
🚨 [HIGH] sequence_anomaly: Novel transition: 'search_web' → 'delete_file'
(never observed; known transitions: ['parse_document'])
--- Injecting: Output drift on 'write_response' ---
🚨 [CRITICAL] token_anomaly: 'write_response' token_count is 2000 tokens,
66.5σ above mean (107.8 ± 28.5)
🚨 [HIGH] output_drift: output_length is 3014 chars, 43.3σ above baseline
🚨 [MEDIUM] output_drift: novel structure 'code_block' (seen: ['plain_text'])`
The latency spike at 76σ. The delete_file call never seen before. The token count 18x baseline. All caught automatically.
Design Decisions
- Zero external dependencies — core only requires numpy, no embedding models or network calls
- Per-tool baselines — each tool gets its own statistical baseline
- Non-blocking — never crashes your agent, errors go to stderr
- Framework-agnostic core — thin adapters for each framework, LangChain first
What's Coming
- CrewAI and OpenAI Agents SDK support
- Persistent baselines across runs
- Slack / PagerDuty alerting
- Hosted dashboard for teams
Try It
bashpip install drift-detection
GitHub: GitHub: dombinic/Drift
Star it if it's useful. Open an issue if you want a feature — I read everything.
Top comments (1)
The "logs show zero errors while the agent does the wrong thing" framing nails the gap — traditional observability assumes failures throw, and agent failures mostly don't. The Markov sequence detector is the part I'd push on. "This tool path has never been seen" is a great signal for genuinely dangerous calls (a
delete_fileout of nowhere), but agents that are supposed to explore — research agents, anything with open-ended planning — will trip it constantly with novel-but-correct paths, and the baseline keeps drifting every time you tweak a prompt. The crux isn't detecting novelty, it's separating novel-and-fine from novel-and-broken.What worked for us was using cheap statistical detection like yours as a first-stage filter (high recall, tolerable false positives) and then firing a small LLM judge only on the flagged subset to adjudicate — you get the cost profile of z-scores with far fewer false pages. How are you handling baseline drift when the agent's "normal" legitimately changes after a prompt update — manual re-baseline, or a rolling window? That's the operational pain I'd expect users to hit first.