Most teams version-pin their dependencies, read changelogs and get a pull request when a library bumps. Then they hand a chunk of their daily work to an AI service that can change under them with no version bump at all.
This week the Ada Lovelace Institute published a feature called Running up that bill? (Charlie Grossman, 16 November 2026). It catalogues how AI subscriptions have changed what they deliver while the price stayed put. It's written about consumer plans, but it says organisations on business plans "remain exposed to practices like plan repricing or changes to how usage is counted."
Here's the engineering version, and a small fix.
What the feature documents
Some of the examples from the past 15 months, as the feature reports them:
- Anthropic cut Claude Code's weekly limits by 17% after a temporary increase.
- A new Anthropic tokeniser produced up to 35% more tokens for the same text.
- OpenAI cut the reasoning time behind GPT-5.2 Thinking's "standard" setting and kept the name.
- OpenAI's GPT-5 router sent some requests to less capable model variants.
- OpenAI acknowledged an issue that applied the wrong usage limits to 40,000 accounts.
It isn't only pricing games
Two older data points matter more to engineers, because they're about quality, not allowances.
In 2023, Chen, Zaharia and Zou compared the March and June versions of GPT-3.5 and GPT-4. On identifying prime numbers, GPT-4 fell from 84% accuracy to 51%. GPT-3.5 improved on the same task. Their conclusion was that the same named service can change a lot in a short time and needs continuous monitoring.
In September 2025, Anthropic published a postmortem of three infrastructure bugs that intermittently degraded Claude's output between August and early September. At the worst hour, one routing bug hit 16% of Sonnet 4 requests. Anthropic said plainly these were bugs, not demand-based cuts. The bit I'd underline is its admission that "we relied too heavily on noisy evaluations."
If the provider's own evals missed it, "the model feels off today" in Slack is not a detection strategy.
My inference: treat the AI service like a dependency you can't pin
This is my mapping onto engineering work, not something the sources prescribe.
You probably can't pin a consumer chat plan. You can still do the three things you'd do for any upstream you don't control:
- Record the contract. Plan, model names, stated limits, reasoning setting, price. Write it down on the day you adopt the tool.
- Run a canary. A fixed set of your own tasks, run on a schedule, scored the same way every time.
- Keep a change log. One table where canary results and provider announcements sit side by side.
Build the canary set in an afternoon
- Pick 10 to 20 tasks from real work: a function with existing unit tests, a SQL query with a known result set, a log excerpt with a known root cause, a diff with a known bug.
- Prefer tasks with a mechanical pass mark. Tests pass, query matches, bug found. Skip anything that needs a long debate to grade.
- Fix the prompt, settings and context. Store them in the repo next to the expected answers.
- Run it weekly, same day, and record pass count, wall-clock time and tokens where the API reports them. The feature notes that API responses come back with a token count, which consumer web apps don't show.
- Expect noise. The feature cites a 2026 study where the most expensive run of the same coding task typically used around twice the tokens of the cheapest. Look for a shift that holds across two runs, not one bad Tuesday.
If you're on the API, pin a dated model version where the provider offers one, and treat a version change like a dependency upgrade: run the canary before you switch.
The artefact: an AI service change log
Copy this into your repo as ai-service-log.md. The rows are labelled examples to show the shape. They are not measurements from the sources.
| Week | Tool / plan / model | Canary pass (of 20) | Median time | Tokens (API) | Provider change seen | Our change | Action |
|---|---|---|---|---|---|---|---|
| EXAMPLE 2026-W45 | Assistant X, team plan, default model | 18 | 41s | 212k | None | None | Baseline recorded |
| EXAMPLE 2026-W46 | Assistant X, team plan, default model | 18 | 44s | 260k | Changelog: new tokeniser | None | Watch token use, check allowance |
| EXAMPLE 2026-W47 | Assistant X, team plan, default model | 14 | 63s | 255k | None found | Longer system prompt | Revert prompt, rerun before blaming vendor |
| EXAMPLE 2026-W48 | Assistant X, team plan, default model | 14 | 61s | 250k | None found | Prompt reverted | Raise ticket with vendor, attach W45 vs W48 |
Two columns do the real work. "Our change" stops you blaming the vendor for your own prompt edits. "Provider change seen" forces someone to actually read the changelog and status page each week.
What to ask before you renew
Take these to your next renewal:
- How much notice do we get before models, limits, reasoning settings or usage counting change?
- Can we export usage data per user and per request?
- What credit applies if a bug drains our allowance?
Consumer plans give weak answers to all three, the feature argues. Business contracts are where you can get better ones.
Sources
- Running up that bill?, Charlie Grossman, Ada Lovelace Institute, 16 November 2026.
- How is ChatGPT's behavior changing over time?, Chen, Zaharia and Zou, arXiv, July 2023.
- A postmortem of three recent issues, Anthropic, 17 September 2025.
Has your team ever caught an AI tool getting worse with actual numbers, or was it gut feel every time?
Top comments (0)