DEV Community

Cover image for Your AI Model Is a Dependency: Pin It, Keep a Fallback, Re-Verify
Anton Gulin
Anton Gulin

Posted on • Originally published at anton.qa

Your AI Model Is a Dependency: Pin It, Keep a Fallback, Re-Verify

#ai #qa #testing #testautomation

Key takeaways

Treat the AI model in your test pipeline like any other dependency. Pin the exact model ID in one config file. Keep a fallback model that has already passed your real suite. Re-run the same suite on both models on a schedule. When a model changes, disappears, or gets swapped, the switch becomes a one-line change instead of an emergency.

The 19-day proof

Nineteen days. That is how long one of the world's best AI models was simply gone this summer.

Claude Fable 5 went offline on June 12 by government order. It came back on July 1. Between those dates, every test suite that leaned on it had a problem. Some teams fixed it in one line. Others lost days, twice.

The difference was not luck. It was whether they treated the model like what it is: a dependency.

What "the model is a dependency" actually means

Your test suite already has dependencies. A database version. A browser version. A Node version. You pin them all. (Pinning means locking the exact version so nothing upgrades by itself.)

An AI model is the same kind of moving part, with three extra ways to hurt you:

  1. It changes silently. Providers update models behind the same name. The model you tested in March is not always the model answering in July.
  2. It disappears. Deprecations happen every quarter. And as of this summer, so do government-ordered shutdowns. Fable 5 was not deprecated — it was switched off overnight, worldwide.
  3. It gets swapped without telling you. The fine print of the Fable 5 return says blocked requests are rerouted to a different model (Opus 4.8). Your pipeline asks one brain and sometimes gets another.

A test that stands on a part like this, unpinned and unwatched, is not a test. It is a hope.

Rule 1 — Pin the version

Never let a test suite float on "latest" or on a bare model family name. Point it at the exact model ID, in one place.

// config/models.ts — ONE place the whole suite imports from
export const MODELS = {
  primary: "claude-fable-5",      // pinned: the exact ID we validated
  fallback: "claude-opus-4-8",    // pinned: the exact ID we validated
} as const;
Enter fullscreen mode Exit fullscreen mode

Every agent, every AI-assisted test imports from this file. Nothing names a model directly. When the world changes, you edit one line, not forty files.

If you use an agent framework, the same rule applies to its config. In Stagehand, for example, the model is an explicit setting — set it, do not rely on an implicit choice:

const stagehand = new Stagehand({
  modelName: MODELS.primary,   // never omit this
});
Enter fullscreen mode Exit fullscreen mode

Rule 2 — Keep a fallback you have already validated

A fallback you pick during the outage is not a fallback. It is a gamble made under pressure.

Pick the second model now, on a calm day. The bar is simple: it must run your real suite acceptably. Not "it is a good model" — your suite, green, at a cost you accept.

Write it into the same config file (see MODELS.fallback above). One line to switch, pre-approved, pre-tested.

Rule 3 — Re-run the same suite on both

This is the rule most teams skip, and it is the one that makes the other two real.

A fallback is only valid while it stays valid. Models drift. So, on a schedule, run your same test suite against both models and compare:

// parity.spec.ts — the same checks, both brains
for (const model of [MODELS.primary, MODELS.fallback]) {
  test(`checkout flow passes on ${model}`, async () => {
    const agent = makeAgent(model);
    await agent.run("complete checkout for the seeded cart");
    // The fixed check the agent cannot move — same for both models:
    expect(await getOrderTotal()).toBe(41.97);
  });
}
Enter fullscreen mode Exit fullscreen mode

Two details carry this pattern:

  • The assert is model-independent. The known answer (41.97 for a seeded cart) comes from your test data, not from the model. Either brain must reach the same truth.
  • Run it weekly, not "someday." A parity suite that ran last night makes a model switch a non-event. A parity suite that never runs makes it an emergency.

What this cost the two kinds of teams

When Fable 5 went dark, the pinned teams with a validated fallback did this: change one line, run the suite, ship. When it came back July 1, they did it again in reverse. Two boring mornings.

The floating teams debugged broken builds twice in three weeks — once out, once back. Same event, same tools. The whole difference was written before the crisis: one config file, one spare model, one recurring suite.

The checklist

  • [ ] Every model reference goes through one config file
  • [ ] Model IDs are exact and pinned — no "latest", no bare family names
  • [ ] A fallback model is chosen, and it has passed your real suite
  • [ ] A parity suite runs both models on a schedule
  • [ ] A model switch is documented as a one-line change anyone can make

You would not float your database version in production. Your AI model earns the same respect — it is infrastructure now, and this summer proved it can vanish like infrastructure too.

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

Top comments (0)