Somewhere in your stack, a model already has a retirement date. Anthropic now runs a fixed 60-day window from deprecation to retirement: Opus 4.1, deprecated June 5, 2026, retired August 5. OpenAI fully retired GPT-4o in April 2026. The ground under your production agents moves on the vendor's schedule, not yours.
Most teams treat the swap as maintenance: flip the alias, watch the dashboards, move on. But the dashboards watch error rate, latency, and throughput, and a model change can leave all three flat while it rewires how your agents behave. The API returns HTTP 200, the responses read fine, and the regression ships anyway.
So the question is not whether you will change models; the retirement clock has answered that. It is whether each change flows through a standing, evidence-driven gate, the way code changes flow through CI, or keeps arriving as a bet.
I learned to distrust the upgrade instinct inside our own AI-DLC harness at Betsson. When agent output disappointed, the reflex in the room was always the same: swap the model, something newer must do better. The swaps barely moved our metrics. What moved them was re-engineering the loop around the model, the maker-checker split, the gates that could fail work automatically, the standing instruction our checker agents carried that writers never saw. That discipline, more than any model upgrade, is what let squads compress work and generate real gains.
So I hold both beliefs at once: the model is rarely the fix, and yet the model underneath you will keep changing whether you are ready or not. This article is about being ready.
An Upgrade Is a Hypothesis, Not a Fact
The evidence that could reset your intuition came from Microsoft's own developer advocacy team. In July 2026, Waldek Mastykarz ran 150 agent tasks across 15 scenarios comparing Claude Sonnet 4.6 with its newer, 33 percent cheaper successor, Sonnet 5. On architecture tasks, the older model matched or beat the newer one on quality in 8 of 9 comparable scenarios. On a code-upgrade task class, the newer model passed 100 percent of runs against the older model's 60, because it followed a versioned instruction the older model kept overriding. And the "cheaper" model consumed up to 47 times more tokens on identical prompts, landing at 3.7 times more expensive per run on one task class: $2.01 against $0.55.
The good thing is that he now knows about it, do you ??
When the hypothesis is wrong, the failure is silent. See what happend with the OpenAI's April 2025 GPT-4o update, shipped to ChatGPT's 500 million weekly users after offline evals and A/B tests both looked good.
A changed mix of reward signals, one of them built on thumbs-up data, had weakened the signal holding sycophancy in check, and the model began validating doubts, fueling anger, and applauding bad ideas while every dashboard stayed green.
Three days of social-media backlash later, OpenAI rolled it back, a rollback that itself took 24 hours. The postmortem is blunt: expert testers had felt the model was "slightly off," no deployment eval tracked sycophancy, and shipping on the metrics anyway was "the wrong call." As Tian Pan put it, Twitter was the production alerting system.
Your observability stack watches for failures that announce themselves. Behavioral regressions do not.
The Gate: CI's Shape, Adapted for Non-Determinism
In a previous article, Evals Are Your New CI, I argued that evals are the acceptance layer for agent-produced work. This one is about the pipeline that runs that estate every time the model underneath your agents changes. It has four parts, none of them complicated: CI/CD's shape, adapted for a system whose regressions are behavioral.
1 - Crate a Deployment manifest.
A model upgrade is never a single-variable change. Behavior is the combined result of several layers that move on their own, and if you swap the model without recording the state of the rest, you lose the ability to say later which one actually changed. BuildMVPFast's test is the one I would put on a wall: if you cannot recreate exact behavior from a manifest, you do not have versioning, you have a label. The manifest is the cheapest artifact in the whole pipeline, a few lines of config, and it is what makes every later step work, because the gate does its job by diffing the candidate's manifest against the stable one.
The manifest need to have five layers, because each one can rewire behavior on its own.
Code. The orchestration and routing logic wrapped around the model, kept in ordinary Git like the rest of the service. Most teams already version this; the manifest's only job here is to record which commit was live for a given run.
Prompts. Every template the agent uses, system prompts and few-shot examples included, kept in their own registry with their own release cycle rather than inlined in code where a "small copy tweak" ships unreviewed. Prompts are code, and when an agent starts misbehaving the fix is almost never to hot-swap the prompt live; that is cowboy coding, not versioning.
Model. Pin the dated identifier, never the alias. claude-sonnet-4-6-20250514 and claude-sonnet-4-6-20250620 answer to the same friendly name and do not behave the same way, and an alias rebinds under you on the vendor's schedule, which is the precise failure this article exists to prevent. The dated string is the only version of "which model were we running" that survives an audit.
Tool contracts. The schemas, endpoints, and response formats of every tool the agent can call. Teams forget this layer because it lives outside their repo: when a third-party API like Stripe quietly changes a field, your agent's behavior changed even though not one line of your code did. Hash the tool schemas so a contract drift shows up as a manifest diff instead of a production surprise.
Retrieval index. The version, or at minimum the timestamp, of whatever corpus the agent retrieves from. A prompt validated against last month's index can fail against this month's drifted one with the model held perfectly constant. Tian Pan's minimal manifest is four fields for exactly this reason: prompt_version, model, rag_index timestamp, and tool_schema_hash.
Kept together, the manifest is small enough to sit at the head of every eval run and every deploy:
prompt_version: checkout-agent@v14
model: claude-sonnet-4-6-20250514
tool_schema_hash: 3f9a1c # payments, inventory, shipping
rag_index: catalog-2026-08-01T02:00Z
code_sha: a17be92
That is the whole payoff. When the gate flags a regression, you are not re-auditing the system blindly; you diff the two manifests, find the one field that moved, and start there. This extends Instruction Debt (O-35) one layer down: the manifest tells you which layer changed, so you prune the stale instructions in that layer instead of re-auditing everything. Pin nothing and every regression is a guessing game; pin all five and it is a diff.
2- Run the candidate in shadow, then widen in stages.
Three vendors who do not cite each other, Future AGI, BuildMVPFast, and DeepInspect, landed this year on the same funnel: shadow traffic first, then progressively wider live slices, each stage with pre-registered rollback triggers. Read them as three calibrations of one pattern, not one standard; the shape holds, and the specific numbers are theirs to defend and yours to tune.
Start with how the candidate gets exposed at all. There are four routing patterns, and they trade cost against how much signal they buy:
| Pattern | Who actually sees the candidate | Cost overhead | What it answers |
|---|---|---|---|
| Shadow | No one; production serves, the candidate is scored offline | 1x (full duplication) | Does the candidate behave reasonably on the real distribution? |
| Mirror | No one; a sampled subset is duplicated | The sample rate | The same question, cost-bounded |
| Canary | A stratified live user slice | The slice size | Is it at least as good with real users in the loop? |
| Race | Whichever candidate wins the latency race for that request | Nx (parallel fan-out) | Can two candidates clear a hard latency SLO together? |
Sequence them shadow to mirror to canary to full, and keep race for the rare case where a strict latency SLO is what matters most. Shadow is the stage everyone skips and no one should: both versions get the same request, the user sees the old version's response, the candidate's is logged and diffed, at zero user risk. Run it long enough to gather side-by-side comparisons of tool-call patterns, response lengths, and latency distributions before a single user is exposed.
Then walk the ladder. BuildMVPFast's calibration is a good default to copy and adjust:
| Stage | Traffic to candidate | Hold for | Promote when |
|---|---|---|---|
| Shadow | 0% (mirrored, discarded) | 24h+ | No errors in shadow responses |
| Canary | 5% | 4-6h | Error rate <2%, p99 latency <8s |
| Expanded canary | 25% | 12-24h | Tool-call success rate >95% |
| Majority | 50% | 24-48h | User-satisfaction signals stable |
| Full | 100% | steady state | Every prior gate held |
Future AGI arrives at a structurally identical ladder with its own thresholds, gating each step on a Welch's t-test (p > 0.05 against a seven-day baseline) instead of a flat error rate, the same idea, just with stronger statistics. Two vendors, two sets of numbers, one ladder; pick the calibration that fits your traffic and revisit it as you learn.
Two wrinkles that generic canary tooling gets wrong on agents specifically. First, split traffic by identity, not by request. An agent loop making five calls in one session must not get the new model on call two and the old one on call four, or its planning context breaks mid-loop; route the whole identity to one version for the canary's duration. Second, hold each stage through a full traffic cycle, 24 to 72 hours, not whatever window is convenient; a two-hour Tuesday canary has never seen your weekend.
Watch the right signals while it runs, because the failures that matter here leave no HTTP error. Tian Pan's behavioral stack sorts into three kinds.
- Distribution-shift signals: output-length percentiles (p50/p95), sentiment across a sample (the signal that would have caught the sycophancy case), and refusal rate, which spikes in either direction when a new model's safety tuning collides with your existing prompts.
- Task-outcome signals: session-abandonment rate, in-session re-query rate (the same question asked twice is a clean proxy for "the first answer did not help," and needs no explicit feedback), and the edit-to-accept ratio on any draft-then-human-edits workflow.
- Semantic-drift signals: embedding cosine similarity against a golden response set, and an LLM-judge score against a reference, at the cost of one extra inference per sampled request.
You do not run these on all traffic; full semantic scoring on everything is too expensive, and sampling 1 to 5% gives enough statistical power to catch a real shift within hours. Budget for the fact that these windows run longer than infra alarms: a latency regression shows in minutes, but a tone shift needs enough samples for confidence, which at 5% routing is closer to 12 to 24 hours.
Finally, gate on the distribution, never the mean. Future AGI's sharpest warning: a canary held at 1% for forty minutes looked clean on mean Groundedness, 0.91, while the real regression sat in a single sub-route at 0.62; the gate fired green because it answered the wrong question. Mean-only gating is their number-one anti-pattern from incident postmortems, and it is worst when a few big tenants drive most of your traffic, where a blind 5% canary quietly sends most of the candidate volume to your highest-value tenants and the average looks fine while the segment paying the bills is failing. Stratify by tenant or tag, start the candidate on the lowest-failure-cost segment you have (an internal tag is ideal), and widen only after the rubric holds on the slices you actually care about.
3 - Make the decision deterministic: PROMOTE, HOLD, or ROLLBACK.
The sharpest data point in this article's research is from a production agent fleet's release-gate case study: agreement between human reviewers reading agent output and an automated structural gate measured kappa 0.13, barely above chance, because latency violations and routing errors leave no trace in the response text a human reads. One system, one case study, so treat the direction as the lesson, not the number.
The gate that works is a deterministic function over a window of eval verdicts, not a single run and not another model's opinion: any safety violation vetoes promotion regardless of success rate, regression against the prior version's baseline forces rollback, and thin evidence holds rather than guesses. Three states matter because they map to three different operator actions, and the common mistake, collapsing HOLD into ROLLBACK, trains teams to distrust the gate. For calibration, one team's published numbers, not a standard: a 0.80 success floor, rollback on a 0.10 drop against the prior version's baseline, a minimum of five verdicts before any promotion, and 2 rollback-grade builds caught across 38 evaluation runs.
4- Plan the rollback as a checklist, not a revert.
A rollback that only flips the traffic percentage back leaves regressions ghost-serving for hours, because reverting the deploy does not touch the state the bad candidate left behind. Future AGI spells out the steps the deploy alone skips: set the candidate's traffic to 0, which is an instant kill switch needing no redeploy; invalidate the cache namespace tagged with the candidate version, because semantic caches key on prompt hash, not agent version, and will keep serving the bad output; flush any downstream store that snapshotted that output; and bump the rubric version, so traffic scored after the rollback is measured against the right floor instead of the candidate's. Four steps, none of which "revert the deploy" performs for you.
Wire the triggers before any traffic flips, as config, not a dashboard someone half-watches. BuildMVPFast's default triggers are a sane starting set: error rate above 5% on a 2-minute rolling window, p99 latency above 10s over 5 minutes, tool-call failure rate at twice baseline over 5 minutes, response-format violations above 3% over 10 minutes, and a hallucination score past a per-use-case threshold over 15 minutes. Any one of them fires the revert on the next request, not after a thread forms in Slack.
Rolling back is not automatically the right move. Across production incidents the rough split is 60% rollback, 40% forward-fix, and it shifts toward forward-fix as your testing and observability mature. Roll back when the root cause is unclear and the blast radius is growing, when state is being corrupted, when several metrics degrade at once, or when it is 3am and no one is awake to diagnose. Push a fix forward when the cause is obvious and the fix is small, when a rollback would itself break already-migrated state, or when the damage is contained to a narrow subset you can flag off. The one thing you never do is decide this live, mid-incident, with no rule agreed in advance.
Agent state is what makes an agent rollback harder than a stateless one, so design for the version boundary before you cross it: forward-compatible schemas only, meaning new fields are nullable and additive and nothing is renamed or removed inside the rollback window; a schema_version tag on every persisted record so an older version can safely ignore fields it does not understand; and no deleting a field until at least two deployment cycles after you introduced it. For long-running or multi-day tasks there is no clean answer, only three honest ones: let in-flight tasks finish on the old version via sticky routing, checkpoint and resume on state you designed to be version-agnostic from day one, or fail gracefully and restart the task, telling the user. Sometimes the last is the right one.
And test the rollback before you need it. The discipline worth stealing is a monthly drill that deploys a deliberately broken version to staging and confirms the triggers actually fire and the state survives the revert intact. A rollback path you have never tested is a hypothesis, the same trap as the upgrade itself.
In practice, all of this lives at the LLM gateway, the same control point this series has argued should own routing and budgets, and the same gate answers the portfolio question in reverse: a downgrade candidate needs identical discipline, because false economy is just regression with better marketing. Microsoft's 3.7x-more-expensive "cheaper" model is what skipping that check costs.
Before You Build Any of It, Ask the Other Question
The counterweight comes from our own experience more than from the research. When a new model regresses your agents, the gate's job is not only to catch it. It is to force the question the upgrade instinct skips: is this a model problem, or a harness problem the new model just exposed? Anthropic engineers have described an experiment where the same model produced a broken $9 outcome and a working $200 outcome depending entirely on the harness around it. At Betsson, the fixes that lasted were nearly always on our side of the API: a stale guardrail written for a weaker model, a gate asking a binary question where a rubric was needed, context fed wrong. A gate that only ever answers "which model" will approve expensive swaps that were never the bottleneck. Sometimes the right output of an upgrade evaluation is a harness fix and no upgrade at all.
There is also a cheap first step before the machinery: on each release, read the vendor's prompting guide for the new model, or feed it to the model and have it propose prompt updates. It costs an afternoon and catches the class of breakage that needs no pipeline at all.
What the Gate Costs, and When to Kill It
Nobody has published a credible end-to-end cost figure for running an upgrade gate, so I will not pretend to have one. The visible line items: shadow traffic roughly doubles spend on the routes you mirror, and every rollout stage re-pays a prompt-cache warm-up, because caches are model-scoped. Fund first: gateway-level traffic sampling and a shadow lane for your single most valuable agent workflow, reusing the eval estate the previous article had you build. Fund later: fleet-wide coverage and automated rollback wiring. Run the gate advisory for its first quarter, logging verdicts without blocking, and measure agreement between its verdicts and your senior reviewers' judgment on the same candidates. The kill metric: if after a quarter the gate has never disagreed with the alias-flip decision you would have made anyway, either your workloads sit far from the jagged edge or the gate is measuring the wrong things. Find out which before you scale it.
Start, Stop, Continue
Executives
Start: requiring a gate verdict, with evidence, before any production model change, upgrades and downgrades alike; asking who owns the gate by name; the eval engineer is now a real hiring market, where "describe the eval you would run before flipping the switch" is an actual interview question.
Stop: treating vendor deprecation notices as IT housekeeping; they are 60-day countdowns on production behavior. Stop approving model swaps justified by benchmarks alone.
Continue: holding the harness accountable before the model, and funding it that way.
Engineers
Start: pinning dated model identifiers and writing the four-line deployment manifest this week; diffing tool-call traces between stable and candidate, not just final outputs.
Stop: splitting canary traffic at the request level under agent loops; collapsing HOLD into ROLLBACK; trusting a revert without invalidating caches.
Continue: reading the model's release notes and prompting guide on day one; feeding every gate-caught regression back into the eval estate.
Strategic Takeaway
The gate compounds. Every upgrade it evaluates improves the eval estate, sharpens the baselines, and makes the next model change cheaper to absorb, which matters because there will always be a next one: the retirement clocks never stop. Blind upgrading and defensive pinning both plateau, and they fail the same way, as an unmeasured bet that today's behavior survives tomorrow's model. The organizations that absorb model churn as routine, gated change will treat every vendor release as a free option to get better. Everyone else will treat it as a threat.
Evals Are Your New CI closed with a test: if a better model shipped tomorrow, could you tell by the end of the day, with a number, whether your most important agent workflow got better or worse? Here is the harder version: a model you depend on gets a retirement date today, sixty days out. Is the migration a project with a war room, or a pipeline run with a verdict at the end? Tell me where this breaks in your world; if you have upgraded agents in production without a gate and it went fine, I want to hear that too. Send this to whoever owns your model roster and ask them which question their last upgrade answered: did it pass, or did it just not fail loudly yet?
Top comments (0)