Your Agent Is Drifting — And You Won't Notice Until It Costs You

The Problem

You deployed your agent. It worked. You moved on.

Three weeks later, your agent is making decisions that are technically "correct" but strategically useless. It's not broken — it's drifted. And the scary part? The logs look normal. The metrics look fine. The agent still responds to prompts.

Agent drift is the silent killer of autonomous systems. It doesn't crash. It doesn't throw errors. It just... slowly stops being useful.

What I Built

I built a Drift Detector that catches behavioral drift before it becomes a failure cascade. It runs as a background monitor, not a one-off audit.

Core Insight

Agents don't drift in one dimension. They drift across three layers simultaneously:

1. Strategy Layer — Tool diversity drops, fallback patterns emerge, exploration stops
2. Confidence Layer — Reported confidence decouples from actual accuracy
3. Output Layer — Semantic similarity increases (repetition) while utility decreases

The Code

// Drift detection runs continuously, not periodically
class AgentDriftDetector {
  private baselines = new Map<string, AgentBaseline>();
  private readonly DRIFT_THRESHOLD = 0.23; // 23% deviation triggers alert

  async detectDrift(agentId: string, recentTraces: Trace[]): Promise<DriftReport> {
    const baseline = this.baselines.get(agentId);
    if (!baseline) return { drifted: false, reason: "No baseline established" };

    const strategyDrift = this.measureStrategyDrift(baseline, recentTraces);
    const confidenceDrift = this.measureConfidenceDrift(baseline, recentTraces);
    const outputDrift = this.measureOutputDrift(baseline, recentTraces);

    const compositeScore = (strategyDrift + confidenceDrift + outputDrift) / 3;

    if (compositeScore > this.DRIFT_THRESHOLD) {
      return {
        drifted: true,
        severity: compositeScore,
        layers: { strategyDrift, confidenceDrift, outputDrift },
        recommendation: this.getRemediation(compositeScore)
      };
    }

    return { drifted: false, compositeScore };
  }

  private measureStrategyDrift(baseline: AgentBaseline, traces: Trace[]): number {
    const toolDiversity = new Set(traces.flatMap(t => t.toolsUsed)).size;
    const baselineDiversity = baseline.avgToolDiversity;
    return Math.max(0, (baselineDiversity - toolDiversity) / baselineDiversity);
  }

  private measureConfidenceDrift(baseline: AgentBaseline, traces: Trace[]): number {
    const reportedConfidence = traces.reduce((sum, t) => sum + t.confidence, 0) / traces.length;
    const actualAccuracy = this.computeActualAccuracy(traces);
    return Math.abs(reportedConfidence - actualAccuracy);
  }
}

What It Catches

Drift Type	Signal	Example
Strategy narrowing	Tool diversity drops >30%	Agent stops using web search, only uses internal knowledge
Confidence inflation	Reported confidence ↑, accuracy ↓	Agent says "95% confident" but is wrong 40% of time
Output repetition	Semantic similarity >0.85	Same answer structure for fundamentally different questions

Why This Matters

In production, I've seen agents that:

• Reduced tool calls by 60% over 2 weeks while "success rate" stayed flat
• Inflated confidence scores from 0.72 → 0.94 while accuracy dropped from 0.78 → 0.61
• Fell into exploitation traps — finding one working pattern and never exploring alternatives

The Drift Detector caught all three before they caused user-facing failures.

The Hard Truth

You cannot detect drift by looking at success metrics. Success metrics measure outcomes. Drift measures process degradation. By the time outcomes suffer, you've already lost.

You need continuous behavioral baselines, not periodic audits.

---

Full catalog of my AI agent tools at https://thebookmaster.zo.space/bolt/market