Last post ended on a promise: my weekly tally has a rule — two consecutive weeks of model-tag growth is a model problem; one week is weather. This week the rule fired for the first time. Here is the exact protocol I ran, the decision I logged, and what I used to do instead (which never worked).
The two weeks the rule fired
Three weeks of tallies showed model tags scattered at 1, 2, 2 — noise, correctly ignored. Then:
| Week | prompt | fixture | model | data | total |
|---|---|---|---|---|---|
| W4 | 4 | 3 | 3 | 0 | 10 |
| W5 | 5 | 2 | 4 | 1 | 12 |
Model tags climbed two weeks running, and the W5 rejections shared a shape: summaries that used to come back as tight two-sentence extractions were suddenly three paragraphs with a "helpful" intro the contract never asked for. Different prompts, same symptom. That is what model drift smells like — not one prompt failing, but the same new behavior across unrelated steps.
One week is weather. Two is a forecast. Time to run the protocol.
What I used to do (the expensive way)
I used to chase the model. Bad week on gpt-4o-mini → spend a weekend auditioning replacements, port the prompts, re-tune temperature, ship the "new model" — and discover the old model's behavior had been fine all along; what changed was a formatting preference in one system-prompt line I edited three weeks earlier. The audit trail proved it: the drift was my own edit, wearing the model's face.
Swapping models on a bad week is how you trade a known system for an unknown one. The protocol exists to make sure I never do that again.
The protocol: five steps, one evening
Step 1 — Declare a model incident and freeze version bumps. The version-bump gate from three posts back goes on hold. Nobody bumps a prompt version while the model is under investigation, because the contract test can't distinguish "I changed the prompt" from "the model changed the output" when both move at once. One variable at a time.
Step 2 — Run the golden fixture suite against both behaviors. The fixture files from the prompt-contract test give me a fixed yardstick. I re-run the full suite on the exact same fixtures, but I split the runs: half forced through the pinned old model behavior (temperature 0, same system prompt, recorded baseline outputs from last month's suite run), half through the current model. Same fixtures, same prompts, two model behaviors. The tally tags came from rejected outputs; this gives me controlled comparisons instead of vibes.
Step 3 — Diff the divergences, not the averages. I don't ask "is the new model worse on average." I ask: on which fixture IDs does it diverge, and what is the new behavior? This time: 7 of 24 fixtures diverged, and all 7 shared one pattern — the model now prepends a framing sentence before structured output ("Here is a summary of...") even though the contract says "output must start with the JSON object / first sentence is the extraction." The averages were nearly identical. The divergence pattern was everything.
Step 4 — Take one of three decisions. The protocol only allows three outcomes:
- Pin the old model, adapt nothing. Only if the divergence is transient or provider-side noise. Rare. Pinning costs a version lock and a revisit date.
- Adopt the new behavior, update the artifacts. If the new behavior is stable and arguably better, the contract and fixtures bend to it — deliberately, not accidentally. This time: the framing sentences were deterministic across re-runs, so I adopted. The contract gained one line ("strip the framing sentence, then validate"), two fixtures were re-baselined, and the n8n step gained a one-node cleanup.
- Quarantine the step. If the divergence is real but unpredictable, the step goes back through the human review queue every run until the behavior settles. Expensive, honest, temporary.
Step 5 — Log the decision with an expiry date. Every model decision goes in a small log: date, evidence (fixture IDs that diverged), the decision taken, and a revisit date. Mine:
{
"date": "2026-10-13",
"evidence": "fixtures F-07..F-13: framing sentence before structured output, deterministic across 3 re-runs",
"decision": "adopt + one-line contract update + 2 fixtures re-baselined",
"revisit": "2026-10-27",
"cost": "one evening, ~2.5 hours"
}
The expiry date matters most. A model decision without a revisit date becomes folklore — "we do it this way because we always did." Two weeks later the log asks: is the behavior still what we adopted? Yes → the decision stays. No → we run the protocol again.
The honest numbers
Two weeks of tallies, one incident, one evening, 2.5 hours, one contract line, two fixtures re-baselined, zero model swaps. The W6 tally is running now; if model tags fall back under two, the protocol worked. If they don't, the quarantine option is still on the table.
Compare with the old way: one weekend auditioning models, a full prompt port, a week of new unknowns — and the actual bug was never the model. The protocol's real output isn't the decision. It's the one I didn't make: no model swap, no temperature fiddling, no "maybe try the other provider." The tally earned the right to touch the model knob, and the protocol spent that right carefully.
Where this sits in the stack
Five layers now, each with one job: the pre-publish script owns the mechanical contract, the review queue owns prose quality, the tally owns the direction of blame, the contract test owns prompt-fixture agreement, and the model-change protocol owns model drift. When the next provider update ships quietly on a Tuesday, I won't notice it by vibes. I'll notice it in the tally, in the fixture diffs, in a log with a revisit date.
Next in this series: the model decision log after a quarter — which decisions aged well, which ones the revisit date killed, and what the log taught me about my own optimism.
I'm building AI Automation Lab — n8n workflow templates for AI automation — and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.
Top comments (0)