When an operating system deprecates an API, you get a year's notice, a migration guide, and a conference talk. When a model provider retires a model, you get a blog post, a date a few months out, and a designated replacement — allegedly better. Our AI features run on the most perishable dependency anywhere in the platform.
We haven't been forced through an emergency swap in production yet. What we have done, constantly, is try models — a lot of them. And somewhere along the way, each experiment stopped being a test and became a rehearsal: the day a retirement notice lands on a model we depend on, the move it triggers is one we've already made dozens of times.
Here's what makes that move routine instead of a crisis.
A model is a catalog line
In our generation pipeline, a model isn't a choice wired into the code. It's an entry in a catalog: which provider serves it, which key unlocks it, what it costs. The catalog covers essentially every major provider — OpenAI, Anthropic, Google, etc. — and production runs a deliberately trimmed selection of it.
The other half of the setup: each step of the pipeline is mapped to its own model. The step that plans doesn't need the model that writes; the step that repairs a single field doesn't need the model that generates a whole component. So a swap has a scope. Changing the model behind one step is a configuration change with a bounded blast radius — not a migration.
The prompt faces the model. The evals face us.
Two kinds of documents govern the model's work, and they look in opposite directions.
The prompt faces the model. It speaks the model's language, and when we try a new family of models, adapting it is real work — but marginal work. Each family has its dialects: how it likes constraints phrased, what it does with structure, where it needs repetition. Moving between them is translation, not renegotiation. The contract — what must come out: the schema, the constraints, what to do when a request is out of scope — doesn't move.
The evals face us. This is the part that took us the longest to see clearly: an eval suite that tries to anticipate what a given model will answer is doing the prompt's job, badly. Ours validate the output against our standard — does this parse, does it respect the schema, does a conversational question get a conversational answer instead of a broken build — and that standard is, by construction, common to every model. The test fixtures come from real production failures. The thresholds are written down. And nothing in them mentions a model by name.
That's the whole trick, and it fits in one sentence: the prompt is bilingual; the standard isn't. Swapping a model means changing who you talk to — not what you accept.
The customer holds the same dial
There's a simple way to check whether a component is genuinely swappable: see who you're willing to let swap it.
On our RAG chatbot, the customer picks the tier of model it runs on — a light one for high-volume answering, a more capable one when nuance justifies it. Their content, their audience, their bill: their dial. We could not hand that dial to thousands of customers if changing the model changed what the feature is. The feature is the retrieval, the access rules, the grounding in their content. The model is staffing.
"Better" is also a regression
The trap in every swap isn't the new model being worse. It's the new model being different — smarter, even — and answering in ways the old contract never anticipated. More fluent refusals. Cleverer formats nobody asked for. A model that upgrades your feature's behavior without your consent has broken it, just politely.
Which is why the plan for retirement day is deliberately unheroic: add a catalog line, adapt the dialect, run the harness, read the thresholds. A component that never answers the same way twice taught us to treat the model as a vendor under contract. This is the other half of that discipline, the one you only see over time:
We don't upgrade models. We re-certify them.
Top comments (0)