A Checklist for Rotating Out a Deprecated Model String

#ai #machinelearning #tutorial #llm

A deprecation notice gives you a date and a replacement string. The work is not the replacement; it is finding every copy, including the copies that are not string literals and the copies that live in your database rather than your repository.

Find it, including where it is not a literal

Start with the obvious search, but search for the family rather than the exact string. A rotation from one snapshot to another will not surface the place that hardcodes a different snapshot of the same model, and that place is on the same deprecation clock.

# Not just the string you were told about — the family, everywhere.
rg -n --hidden -g '!node_modules' -g '!.git' \
   -e 'gpt-4[o0.-]' -e 'claude-[a-z0-9-]*-2[0-9]{7}' -e 'gemini-[0-9]' .

# And the assembled ones, which the search above cannot see:
rg -n -e '"gpt-" *\+' -e "f\"gpt-" -e 'MODEL_PREFIX' -e 'model_name *=' .

The second search matters more than the first. A model identifier built by concatenation — a base name plus a tier suffix from config, or an f-string interpolating a family and a date — is invisible to a literal search and is exactly the kind of code that ends up choosing a model nobody intended. If you find one, the rotation is a good moment to collapse it into a single explicit constant per model. A name assembled at runtime cannot be checked at build time and cannot be found by the next person doing this.

Nine places a model string hides

In rough order of how often each is the one that gets missed:

The fallback chain. The primary is what everybody edits. The second and third entries are what everybody forgets, and they are only exercised when the primary is already failing — so the bug lands during an incident, which is the worst possible time to discover that your degraded path 404s.
Stored rows, not just code. Anywhere a model is selected per tenant, per workspace or per saved configuration, the string is in a database column and no repository search will find it. This needs a data migration, and it is the item most likely to turn a one-hour rotation into a week.
Seed data and factories. Test factories, demo fixtures and onboarding defaults create new rows carrying the old string, so a data migration that runs on Monday is undone by every signup on Tuesday.
Recorded HTTP fixtures. Cassettes and snapshot responses match on the request body. Rotating the model in code without re-recording produces a test suite that fails for a reason unrelated to behaviour, and the usual fix — loosening the matcher — quietly removes the check that the right model is being called at all.
Eval and regression configuration. A harness still pinned to the old snapshot keeps passing after the rotation and is no longer testing production. See running a prompt set across two versions.
Cost and pricing tables. Per-token rates keyed on the model identifier. Miss this and the requests succeed while every cost figure you report is wrong — often silently zero for the unknown key.
Observability config. Dashboard queries, alert rules and log filters that select on the model label. These do not break the application; they break your ability to watch the rotation, which is worse, because the graph goes flat and looks like success.
Infrastructure and CI. Terraform, Helm values, GitHub Actions environment blocks, Dockerfile defaults, and the deployment name on Azure — where, as covered in model name strings, the rotation may be a portal change rather than a code change.
Documentation, SDK snippets and support macros. Not load-bearing for the service, load-bearing for every integrator who copies from them.

The rotation, in order

The order exists to keep the system runnable at every intermediate commit. Doing the data migration before the code accepts both values is the usual way to take an outage.

Introduce one constant. Define the model in a single module and make every call site read it. This is the change that makes every subsequent rotation cheap, and it is safe to ship on its own with no behaviour change.
Make the code tolerant of both strings. Pricing tables, capability checks and metric labels should recognise the old and the new identifier before either the code or the data moves. Skipping this is what makes step 4 irreversible.
Rotate the default in code, behind a flag or a small percentage. The blast radius of a wrong model is a quality regression, not an error, so ramp it the way you would ramp a behavioural change and not the way you would ship a config edit. See canary releasing a model.
Migrate stored rows. A single UPDATE mapping old to new, plus the seed data and factories in the same commit so new rows cannot reintroduce the old value.
Rotate the fallback entries. Then force the fallback path in a test environment and confirm it serves rather than 404s. A test that exercises the order is the only way this stays true.
Re-record fixtures and update dashboards. Do these together: the fixtures prove the request changed, the dashboards prove you can still see it.
Remove the old value. Only after the metric for requests carrying the old identifier has been flat at zero for longer than your longest cache or queue delay.

Verifying the old string is gone

A repository search that returns nothing proves the repository is clean, not the running system. The check that actually answers the question is at the egress: emit the model identifier from the response body — the resolved one, not the requested one — as a label on your request metric, and query for the old value over a window longer than your slowest scheduled job. Nightly batch work, retry queues and per-tenant overrides all surface here and nowhere else.

Two failure modes to look for specifically. A count that drops to a small nonzero plateau is almost always stored configuration you did not migrate. A count that drops to zero and returns after a deploy is seed data or a factory recreating rows.

Stopping it coming back

After the rotation, add the cheap guard rather than the elaborate one. A lint rule or a CI grep that fails on any model-shaped literal outside the one constants module catches reintroduction at review time, costs nothing to run, and is a rule reviewers can enforce without reading the deprecation notice.

The complement is a startup assertion: on boot, resolve every model identifier the service can select — primary, fallbacks, per-tenant overrides — against the provider’s list-models endpoint, and refuse to start, or log loudly, if any of them is absent. That turns the next expiry into a deploy-time failure in a staging environment rather than a production 404 at the moment the provider flips the switch. Both providers publish deprecation schedules ahead of time — OpenAI maintains a deprecations page and Anthropic publishes model deprecations in its documentation — but a schedule you have to remember to read is not a control.

Deprecation notice periods differ by provider and by model tier, and preview or experimental snapshots are typically excluded from whatever notice the stable ones receive. Read the current schedule rather than relying on a remembered window.