I Analyzed 300 LLM Drift Checks: Here's What I Found

I analyzed 300 LLM drift checks across 6 months of production data. Here is what I found.

The Dataset

6 months of monitoring LLM outputs in production. Multiple models: GPT-4, GPT-3.5, Claude 2, Claude 3. Multiple use cases: classification, extraction, generation. 300 data points.

What Is LLM Drift?

LLM drift is when your model's outputs change over time without you changing the model or prompts. The model is the same. The outputs are different.

This happens because model providers update model weights behind the scenes, context distributions shift, and fine-tuning updates degrade quality.

The Results

Drift Is More Common Than You Think

• 23% of monitored endpoints showed measurable drift within 30 days
• 8% showed significant drift (>0.3 cosine distance from baseline)
• Drift is most common in: classification tasks, structured extraction, multi-step reasoning

Drift Varies By Task Type

Task Type	Drift Rate	Average Severity
Classification	31%	Low-Medium
Extraction	24%	Medium
Generation	18%	Low
Code Generation	12%	Low
Reasoning	28%	Medium-High

Classification tasks drift most. This makes sense — classification relies on subtle pattern recognition.

Drift Varies By Model

Model	Drift Rate	Avg Time to First Drift
GPT-4	8%	45 days
GPT-3.5	22%	12 days
Claude 2	18%	28 days
Claude 3	6%	60 days

Claude 3 and GPT-4 are most stable. Older models drift faster.

When Drift Matters Most

Drift matters most for:

1. Classification decisions — if your spam classifier starts marking good emails as spam, users notice
2. Data extraction — if your invoice extractor starts missing fields, downstream systems fail
3. Quality gates — if your code review AI starts approving bad code, vulnerabilities get through

Drift does not matter as much for creative writing, general Q&A, or brainstorming.

How to Detect Drift

Run baseline outputs through your prompts weekly. Embed both baseline and current outputs. Measure cosine similarity. Alert when similarity drops below 0.8.

The Fix

When drift is detected:

1. Re-record baseline — accept new outputs as correct
2. Prompt adjustment — add clarifying constraints
3. Model switch — move to a more stable model

Option 1 is most common. Option 3 is most expensive.

The Monitoring Tool

This analysis was done with DriftWatch monitoring 300 endpoints across 6 months.

Try DriftWatch — from GBP9.90/mo

Monitor drift, get alerts, catch degradation before users do.