Someone flips a feature flag back on to compare two behaviours, and the request fails immediately with a not-found error naming a model string nobody has typed in eleven months. The code is current, the deploy is current, the flag is old — and flag configuration is the one place your migration never looked.
The error, months after the change that caused it
The literal string differs by provider but the shape is constant. On Anthropic’s API a retired or misspelled model id returns HTTP 404 with an error object of type not_found_error; the publisher documents typos and deprecated ids as the two usual causes (platform.claude.com, errors). OpenAI-compatible endpoints return a 404 whose message follows the form “The model ‘…’ does not exist or you do not have access to it”. Either way the string that lands in your log is a model id that is not in your source tree, which is what makes it confusing: grep across the repository finds nothing.
Two properties make this failure mode nastier than a plain 404 in application code. It is latent — the flag variant was serving zero percent of traffic, so nothing exercised the path between the model’s retirement and the toggle. And it is attributed to the wrong change — the person who flipped the flag will reasonably believe they broke something, and will spend the first twenty minutes looking at the wrong diff.
Why flag payloads escape every other audit
A migration checklist is built around the code that calls the model. People grep the repository for the old id, update the constants, fix the tests, and ship. Flag configuration is not in the repository — it is in a flag service’s database, or a JSON blob in a config store, or a remote-config document. It is data, not code, so it is invisible to the static search that drives the rest of the migration.
Worse, the value is usually not tagged as a model id at all. It is a string inside a variant payload called value or config, alongside colour codes and copy strings. Nothing in the flag system knows it names an external resource with a retirement date attached. Model ids also leak into the same class of unscanned surfaces elsewhere: environment variables in a deploy manifest, a column in a per-tenant settings table, a saved prompt template with a pinned model, a scheduled job’s arguments.
There is also a timing trap. Providers usually announce a retirement well in advance and keep serving the id until the stated date, so the window in which a flag payload is wrong but still working can be months long. Anything you check during the migration itself passes. The check has to run after the retirement date, on a schedule, which is why the audit below belongs in a nightly job rather than in a migration runbook that gets archived when the migration closes.
Where retired ids hide
- Percentage-rollout variants at 0%. The classic. A model-comparison flag left in place after the experiment concluded, with the losing arm still configured and still naming the old model.
- Per-tenant overrides. One customer pinned to a specific version during an incident, with a note that says “temporary”. Only that tenant’s requests fail, so error rates barely move and alerting does not fire.
- Kill switches and fallbacks. A flag whose
offstate is meant to route to a known-good older model. This one is genuinely dangerous: the fallback fails exactly when you reach for it, during an incident, which converts a degradation into an outage. - Archived experiments. Flags marked inactive in the interface but whose payloads are still fetched by a client that does not check the active bit.
The audit job
The check is straightforward once you accept that flag data is a surface you must scan. Enumerate every variant payload of every flag, extract anything that looks like a model id, and cross-check it against the provider’s live list of models. Run it in CI nightly, not only at migration time.
# audit_flag_models.py — exits non-zero if any flag names an unavailable model
import re, sys
MODEL_RE = re.compile(r"\b(?:claude|gpt|gemini|llama|mistral)[a-z0-9._-]{2,}\b", re.I)
def strings_in(obj):
if isinstance(obj, str):
yield obj
elif isinstance(obj, dict):
for v in obj.values():
yield from strings_in(v)
elif isinstance(obj, list):
for v in obj:
yield from strings_in(v)
available = {m.id for m in client.models.list()}
problems = []
for flag in flag_service.list_flags(include_archived=True):
for variant in flag.variants:
for s in strings_in(variant.payload):
for candidate in MODEL_RE.findall(s):
if candidate not in available:
problems.append(
f"{flag.key} / variant {variant.key}: "
f"{candidate!r} is not in the provider's model list"
)
for p in problems:
print(p, file=sys.stderr)
sys.exit(1 if problems else 0)
Two design choices matter. Pass include_archived=True: an archived flag whose payload is still served is precisely the case that bites. And compare against the provider’s live list rather than a hard-coded allow-list of current ids, because a hard-coded list is another string that goes stale and you are back where you started. Where a provider offers no list endpoint, the fallback is a startup probe — issue one minimal request per distinct id found and treat a 404 as a failure.
Model ids appearing in flag payloads are also a signal worth acting on independently of retirement. Every id in flag data is a routing decision made outside your routing code, and it will diverge from the selection tree eventually.
Flag hygiene that prevents the next one
Three practices remove the class rather than the instance.
Store an indirection, not an id. Put "model_role": "summarizer_v2" in the flag payload and resolve the role to a concrete id in code. The flag now expresses intent, the id lives in one place that a migration greps successfully, and a retirement is fixed by one edit rather than by hunting through a flag console.
Give experiment flags an expiry. A flag created for a model comparison should carry a removal date in its description and appear on a stale-flag report after it. The audit above catches the symptom; deleting concluded experiments removes the surface.
Exercise the off path. If a flag exists so you can fall back during an incident, that fallback needs to be exercised on a schedule, in the same way you would test a database failover. A synthetic request per flag variant per day, asserted on for a 2xx, turns a latent 404 into a routine alert weeks before anyone needs the switch.
One more thing worth doing while the failure is fresh: make the error message say where the id came from. A log line reading “model not found” sends someone to the repository. A log line reading “model not found; id supplied by flag summarizer-arm-b, variant control” sends them to the flag console, which is where the problem is. That requires threading the provenance of the model id through to the call site, which is a small change and pays for itself the first time somebody flips an old switch on a Friday.
Top comments (0)