DEV Community

Khavel
Khavel

Posted on • Originally published at aimodelwatch.dev

Make your CI fail before your model does

#ai #llm #devops #github

Today, August 5 2026, claude-opus-4-1 retires. Anthropic said so on June 5, on their model-deprecations page, with the date in plain text. Two months of notice.

If that id is hard-coded in a repo somewhere, nothing in the build system found out. Tests passed all summer. The linter had no opinion. The dependency bot, which will open a PR because a transitive dev dependency moved from 4.2.1 to 4.2.2, has no idea the model your product is built on has an expiry date on someone else's calendar.

That is the actual shape of the problem: a model dying is a calendar event, not a code change. Nothing in CI watches a calendar.

Why your provider's email doesn't cover you

It does exist. Anthropic, OpenAI and Google all email deprecation notices — they are contractually motivated to. But they email the account owner: whoever's card is on the org. Not the repo. Not the pull request. Not the person who pinned the id eight months ago and left.

So the notice lands in a billing inbox, and the id sits in src/llm.ts until the call starts 404ing. Between those two events there are usually months where a build could have told you, and didn't.

The numbers, from today's catalog

I maintain AI Model Watch, a daily-verified catalog of model prices, context limits and lifecycle dates, every row carrying the provider URL it was read from. Pulled from the live feed while writing this (updated: 2026-08-05, 204 models):

  • 74 of the 204 are already deprecated or retired.
  • 12 carry a retirement date inside the next 90 days (13 if you count the one retiring today).

Do not add those together. 9 of the 12 are already inside the 74 — being deprecated and being near your retirement date is the normal case, not two separate populations. The distinct total is 77, not 86. Two counts in one paragraph is a set problem, and the word joining them is a claim like any other.

The interesting three are the ones that aren't in the 74. qwen3-max is ga today, priced today, sitting in the current docs looking entirely healthy — and Alibaba's own deprecation table retires it on September 8, pointing at qwen3.7-max. A status field would never have warned you. Only the date does.

Thirty seconds to a check

One file, no dependencies, no key, no signup, Node 18+:

curl -sLO https://aimodelwatch.dev/ci/check-models.mjs
node check-models.mjs --scan .
Enter fullscreen mode Exit fullscreen mode

--scan walks your source and picks out anything that looks like a model id it knows, so there is nothing to configure on the first run. Here is a real run against a three-line fixture — output as it came out, only the Windows path separators normalized:

$ node check-models.mjs --scan .
check-models 1.1.0 — feed updated 2026-08-05, 204 models
scanned . — found 3 model ids in your source
  claude-opus-4-1  src/llm.ts:2
  gemini-2.5-flash  src/llm.ts:4
  qwen3-max  src/llm.ts:3
checked 3 models · retirement threshold 90 days

✖ claude-opus-4-1: Claude Opus 4.1 is DEPRECATED. Retires 2026-08-05 (in 0 days). Provider's stated replacement: claude-opus-4-8.
✖ qwen3-max: Qwen3-Max retires 2026-09-08 — 34 days away (threshold 90). Provider's stated replacement: qwen3.7-max.
✖ gemini-2-5-flash: Gemini 2.5 Flash is DEPRECATED. Retires 2026-10-16 (in 72 days). Provider's stated replacement: gemini-3.6-flash.

3 problems, 0 warnings. Details: https://aimodelwatch.dev/deprecations

$ echo $?
1
Enter fullscreen mode Exit fullscreen mode

Three ids, three providers, three different failure modes: one that expires today, one still marked GA with a date five weeks out, one deprecated with a quarter left. Exit 1 on all of them.

(The id in my source is gemini-2.5-flash and the finding comes back as gemini-2-5-flash — providers write the same model both ways depending on the page, so ids are matched in a normalized form. Worth knowing if you grep the JSON report.)

In GitHub Actions

There is an action, so this is the entire setup — no install step, no setup-node, no inputs:

# .github/workflows/model-lifecycle.yml
name: model lifecycle
on:
  pull_request:
  schedule: [{ cron: '0 7 * * 1' }]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Khavel/check-models-action@v1
Enter fullscreen mode Exit fullscreen mode

With no inputs it walks the repo, finds every model id in the source, and checks all of them. Findings come back as inline ::error annotations on the offending line plus a job-summary table. The action vendors the script, so @v1 pins the code while the catalog stays live.

The cron line is the half that matters. A pull_request trigger only helps on days somebody opens a PR. Deprecations are announced on days nobody opens a PR — that is the entire nature of a calendar event. A weekly run is what turns "we found out in August" into "we found out in June".

The other half: prices move too

Nobody emails you when a model gets cheaper or dearer. There is no deprecation notice for a 40% output-price hike; the invoice just changes shape at the end of the month.

So pin the numbers you costed the product on, and commit the file:

node check-models.mjs --scan . --pin .amw-pin.json --update-pin   # once, then commit
node check-models.mjs --scan . --pin .amw-pin.json                # every build
Enter fullscreen mode Exit fullscreen mode

The pin file is small and diffable — status, the three prices, retires_on, the stated replacement, per model. When a value moves away from what you pinned, it shows up as a build failure with both numbers in it:

✖ qwen3-max: Qwen3-Max input price ($/1M) changed since your pin: 0.8 → 1.2.
Enter fullscreen mode Exit fullscreen mode

(That one I produced by editing the pin to an old value — I am not going to fake a price change to make a point. The mechanism is what's real: it compares your committed snapshot against today's catalog.)

One sharp edge, stated up front

GitHub does not evaluate a composite action's outputs when the action fails the job. So steps.*.outputs.* come back empty on exactly the path you care about — the failing one.

I could not fix that from inside a composite action, so instead the JSON report is written unconditionally: point report-path somewhere and read it with if: always(), or set warn-only: true and branch on the outputs yourself. Worth knowing before you build a Slack notification on top of the outputs and wonder why it posts empty messages.

What's underneath

The check reads a free JSON feed — no key, no signup, CORS open, and every row carries the provider page the values were read from:

curl -s https://aimodelwatch.dev/api/models.json | head -20
curl -s https://aimodelwatch.dev/api/deprecations.json | head -20
Enter fullscreen mode Exit fullscreen mode

Everything in it is what the provider states, taken from its own documentation. Where a provider publishes nothing, the field is null and the check stays quiet rather than guessing — an invented retirement date that fails someone's build would be much worse than a missing one.

If you would rather not put a third-party action in your pipeline, the script is one file with no dependencies and works on any CI. curl it and run it.

Either way: the check costs about two lines, and the thing it catches is a model that stops existing while your tests are green.

Top comments (0)