Wednesday AI Opinion. Here's my hot take: "we use the latest model" is not a strategy. It's a dependency you haven't pinned. Every LLM feature you ship is built on a model that will either change under you or get switched off. Treat the model like any other production dependency: pin it, watch it, and keep a migration eval ready before the retirement email shows up.
The "same" model doesn't stay the same
The classic evidence is Chen, Zaharia and Zou's study How is ChatGPT's behavior changing over time? They compared the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on the same tasks. Their example: GPT-4 (March 2023) identified prime vs. composite numbers with 84% accuracy; GPT-4 (June 2023) got 51% on the same questions. GPT-3.5 moved the other way and got much better in June on that task. Both models made more formatting mistakes in code generation in June than in March.
Their conclusion is the part worth printing out: the behavior of the "same" LLM service "can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs."
If your code calls a moving name, you inherit every one of those changes with no deploy, no diff and no review.
Pinning is easy now. Use it.
Providers give you stable identifiers. Anthropic's models overview says every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on. For models before that generation, the alias is a convenience pointer that resolves to the dated ID.
OpenAI publishes moving variants too. Its deprecation policy lists chat variants such as gpt-5.1-chat-latest as "specialized variants", and preview models (with preview in the name) that can be retired on much shorter notice.
Rule of thumb: in production, send an ID that names exactly one model. Aliases and -latest names are fine for a playground, not for the thing that writes to your customers.
But pinning alone just delays the breakage
Here's the half people skip. A pinned model doesn't live forever.
-
Anthropic: deprecated models get a retirement date, and "Requests to models past the retirement date will fail." Customers with active deployments get at least 60 days' notice for publicly released models. Example from their table:
claude-opus-4-1-20250805, deprecated June 5, 2026, retirement August 5, 2026. - OpenAI: minimum notice is at least 6 months for generally available models, at least 3 months for specialized variants and image models, and preview models "may be retired with much shorter notice, such as 2 weeks." OpenAI says it doesn't recommend preview models for business-critical production workloads unless you can migrate on short notice.
- Real example: OpenAI's past-deprecations table lists
gpt-3.5-turbo(withgpt-3.5-turbo-0125) shut down on Oct 23, 2026, notice date Apr 22, 2026.
So the question isn't if you'll migrate. It's whether the migration is a calm afternoon or a weekend incident.
What a migration costs depends on one thing: your eval set
Both providers say the same thing in their migration guidance. Anthropic: test your applications with newer models "well before the retirement date." OpenAI: "Run representative requests against both models. Compare outputs, latency, and cost before switching production traffic to the replacement."
That's only cheap if "representative requests" already exist as a file in your repo, with a way to score them. Without that, a forced model swap means vibe-checking a handful of prompts and hoping.
The playbook I'd use
- Pin the model ID in config, not scattered across code. One place to change, one place to grep.
- Log the model ID with every request, so when quality moves you can tell a model change from a prompt or retrieval change.
- Keep a migration eval set: real inputs from production (redacted), expected properties or rubrics, plus the edge cases that broke before.
-
Watch for shutdowns programmatically. OpenAI's List models endpoint (
GET /v1/models) has an optionalshutdown_datefield. Anthropic's Console Usage export breaks usage down by API key and model, which is how you find the forgotten service still calling an old model. - Run old vs. new side by side on the eval set before switching: quality, format errors, latency, cost.
- Roll out like a dependency upgrade: canary a slice of traffic, compare, then flip. Keep the old ID in config until the new one has earned it.
- Avoid preview models for anything business-critical unless you've accepted a two-week migration.
The take
Model upgrades aren't free wins and they aren't optional. They're dependency upgrades with a deadline you don't control. Pin the model so changes only happen when you choose, and keep an eval set so choosing is cheap.
How does your team find out a model is being retired: the email, a dashboard, or a failed request?
References
- Chen, Zaharia, Zou: How is ChatGPT's behavior changing over time? https://arxiv.org/abs/2307.09009
- Anthropic: Models overview (pinned snapshots, aliases) https://docs.anthropic.com/en/docs/about-claude/models/overview
- Anthropic: Model deprecations (60 days' notice, retired requests fail, usage audit) https://docs.anthropic.com/en/docs/about-claude/model-deprecations
- OpenAI: Deprecations (notice periods, shutdown_date, migration guidance, past deprecations) https://platform.openai.com/docs/deprecations
Top comments (0)