We wrote a handful of small rules on November 12. Each one was a line or two. Each one was obviously right. A read-only review found a real bug in nearly every one of them before they shipped.
That is not a story about bad rules. I think it is a story about the wrong kind of question being answered with rules at all.
I build Since, which tracks the packages, AI models, APIs and runtimes your code depends on, tells you what changed and when it breaks, and opens the fix pull request. Here is the claim, and then how we got there: when the question is one a person would answer by reading the code, a growing pile of string rules is the bug.
1. The question that looked like string matching
Since needs to know, for every model ID in your code, where the request actually goes. That matters because the same model retires on different dates in different places. claude-sonnet-4-5-20250929 retires on the Claude API on November 30, 2026, and on Amazon Bedrock it runs until April 8, 2027 (both on the live calendar, read today). Get the destination wrong and you tell someone the wrong deadline.
Most of the time this is easy. A string that starts with claude- in a call to Anthropic's client goes to Anthropic. A string that starts with gpt- in a call to OpenAI's client goes to OpenAI.
Then there are gateways and routers. Libraries and services that take one model string and send it somewhere else, each with its own spelling. vertex_ai/.... gemini/.... openrouter/google/.... A dotted Claude version where the provider uses dashes.
So we did the reasonable thing.
2. The rules
A rule for which prefixes mean a gateway. A rule for which ones mean a router. A rule for the dotted Claude version. A keyword rule for vertexai. Each was small. Each matched the example in front of us.
Here is the shape of it, cut down to the idea:
// A rule-based guess at "which provider gets this model id?"
const rules = [
[/^claude-/, "anthropic"],
[/^gpt-|^o\d/, "openai"],
[/^gemini-/, "google"],
[/^vertex_ai\//, "google"],
];
const guess = (id) => rules.find(([re]) => re.test(id))?.[1] ?? "unknown";
for (const id of [
"claude-sonnet-4-5-20250929",
"vertex_ai/gemini-2.5-pro",
"gemini/gemini-2.5-flash",
"openrouter/google/gemini-2.5-pro",
"openrouter/anthropic/claude-sonnet-4.5",
]) console.log(id.padEnd(40), guess(id));
Run in an empty folder with Node, it prints:
claude-sonnet-4-5-20250929 anthropic
vertex_ai/gemini-2.5-pro google
gemini/gemini-2.5-flash unknown
openrouter/google/gemini-2.5-pro unknown
openrouter/anthropic/claude-sonnet-4.5 unknown
Every unknown is a rule we have not written yet. You can see where this goes. Add a rule for gemini/. Add one for openrouter/<provider>/. Add one for the dotted version. Then someone sets the model from an environment variable, or wraps the client in a helper, or uses a prefix that means one thing in one library and another thing in the next.
3. The review
Before releasing, we run a read-only review over the commits that haven't shipped yet. It found a real bug in nearly each rule. Not style. Bugs: a prefix that matched more than it should, an order where one rule shadowed another, a keyword that fired in the wrong context.
What I said when I read that review was about the shape, not the bugs: every gap that doesn't fit the rules needs another rule, and the rules become huge.
That is true of any table of special cases. It is especially true when the thing you're matching is spelled by hundreds of other people's libraries, which change without telling you.
4. The stance
Here is the line we drew, and it is the whole point of this post:
-
Code settles only what it can prove. A literal
claude-sonnet-4-5-20250929passed to Anthropic's own client, in the same file, is proven. Nobody needs to think about it. - Everything in doubt goes to a reviewer that reads the code, the way a person would. It gets the lines with numbers, the client the file actually uses, and one note on what the terms mean. It answers "where does this ID get sent?"
- One general test decides what is in doubt. Not a table of vendors, prefixes or keywords.
- The default stays where nothing is in doubt, so findings that were already right do not move.
- Reading costs money, so it is capped.
Before adding a second special case to any rule now, the question is: can the reviewer we already have answer this with the code in view? Usually, yes.
5. The second implementation was the real bug
The same day, checking the fix on every place Since answers this question turned up something worse than the rules.
The part of Since that checks a pasted snippet had its own smaller matcher. A quoted string equal to a tracked ID, then one platform rule. So a gateway's route got no answer there at all.
Looking for why found the actual core gap. The shared reader knew a route only by a provider's name. So vertex_ai/..., gemini/... and openrouter/google/... were invisible to scans and connected repositories too, not just snippets.
Rules have a way of breeding copies. Someone needs the answer somewhere else, the original is tangled, so they write a smaller one. Now you have two sets of rules that disagree. The fix was not to patch the copy. It was to route the snippet check through the shared reader, delete the copy, and write the agreement as a test: the same code, scanned as a file and checked as a snippet, gets the same places, states, severities and words.
6. We had been here before, in the other direction
This was not the first time rules had been the problem. On November 3, about 40% of our fix attempts were being refused. Most of those refusals came from our own code's gates, each one reasonable in isolation: no checks pass here, a root we couldn't prove, a runner label that wasn't due yet.
What I told my agent then was "we need to stop being super strict... it's refusing a lot of stuff that it should be fixing". So the rule became: code refuses a fix only with proof that the change breaks something (a check that passed before fails after, after one hand-back to the agent), or when nothing at all can check it. Everything else opens, and says plainly what was not checked.
Same disease, opposite symptom. On November 3 the rules refused too much. On November 12 they understood too little. Both times, each rule was defensible and the pile of them was wrong.
The fair counterargument
"So you replaced deterministic code with something that can be wrong in new ways." Yes. That is fair, and it happened to us.
The next day, November 13, a provider redrew its deprecations page and our hand-written reader for it read nothing. The reader that replaced it, one that reads pages the way a person would, was right on that provider's page. Then we ran it on every other page it could ever be given. It read one provider's active models as deprecated. It lost replacements that a page states in a second table. It took a fine-tuned deployment's date for the base model's. None of that showed on the one page we had tested.
So the honest version of the stance has a second half:
- Judgment replaces rules only where rules can't keep up. A literal ID in a known client stays code.
- Judgment gets tested on every input it could see, not the one that motivated it. Build the comparison harness before tuning anything: store the current answers, run the new reader on the same inputs, and nothing may move that shouldn't.
- Judgment may add to what precise code wrote only on proof. If the reviewer is silent, nothing gets erased.
Rules are great at what they can prove. They are terrible at being a model of someone else's code. When the second special case shows up, that's usually the signal you're asking them to be the second thing.
What's the longest list of special cases you've maintained before admitting it was the wrong tool?
I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.
Top comments (0)