Never let a cap decide whether a route exists

I added a per-provider cap to my agent's ranking because one provider kept flooding it. An aggregator with a genuinely large free catalogue was taking most of the visible slots, and what was supposed to be a ranking of routes had quietly turned into a list of one vendor's models in slightly different orders. The cap itself was four lines. The mistake was the sentence I thought right after it: applies to everyone, it's tidier that way. In practice I applied it to every provider.

The list got shorter and stopped being correct

The cap cut that aggregator from 80 free models to 8. In my log that read as a success, because 8 clean entries look like a decision and eighty look like noise. What I did not look at was which eight. The cap did not shorten the list, it decided membership. A live route dropped out because it happened to be entry number nine. A dead route -- the paid-only successor, the one that could never actually serve a request -- stayed in, because it happened to sort earlier.

So the map came out wrong in both directions at the same time, and one of the errors hid the other. Had I only lost a working route, I would have noticed a gap. Had I only kept a dead one, I would have noticed a failed call. Instead the count looked plausible, and the two mistakes cancelled into something that read as normal.

Position is not a signal

The honest part is what the sort was keyed on. Nothing interesting. The order was whatever fell out of discovery: provider order, then model order inside each provider, roughly the sequence the catalogue felt like returning that afternoon. Position was an artifact of iteration order. I then capped on position, which is the same as picking at random, with one extra property that makes it worse than randomness: it is deterministic, so it reproduces, so every run agrees with the mistake.

The order of operations was the bug, and it was structural rather than clever:

routes = discover()
routes.sort(key=sort_key_i_never_questioned)
routes = cap_per_provider(routes, n=limit)   # membership decided here
routes = [r for r in routes if probe(r).ok]  # liveness asked too late

Once a route is removed, nothing downstream knows it existed. There is no exception, no warning, no stale-entry line in the log. The filter that runs afterwards can only confirm what survived; it cannot miss what is gone. That is the shape of the failure I keep running into in agent plumbing: the wrong decision is silent, and every later stage is honest about a list that is already wrong.

Turn the two middle lines around and the same code becomes reasonable. Probe first, sort by what survived and by how well it scored, then trim. A cap at the end of a good sort is a budget. A cap in the middle of a bad sort is a policy about nothing.

The tool that reported its own bug as upstream news

Weeks later the same mistake came back wearing a different hat. My scout reported a route as vanished. I probed the endpoint directly, by hand, and got a 200 OK.

The route had never gone anywhere. My own cap had removed it from the map earlier in the pipeline, and the monitor compared today's post-cap list against yesterday's and reported the difference as an upstream change. The alert was true about my map and false about the world, which is the worst combination available, because it is specific enough to believe and wrong enough to cost an evening. I spent that evening on the provider's status page and then drafting a complaint to nobody. The cause was three functions upstream of the thing I was staring at.

Since then the rule I hold is narrower than "check your assumptions": probe absences before acting on them, and never let a monitor own the word vanished unless it is pointing at a raw endpoint rather than at your own filtered view. A monitor that diffs derived state will report your bug as somebody else's outage, cheerfully, forever.

Where the cap actually belongs

Two things survived from that week, and both are cheap to say and annoying to keep.

Cap only the providers that need capping. The reason I wrote the thing in the first place was local: one catalogue swamping the ranking. Spreading it across every provider was tidiness dressed up as engineering. An aggregator with a genuinely large free catalogue should not share a number with a single-vendor endpoint whose free tier is a handful of models behind a rate limit. One number applied to both is either meaningless on one side or destructive on the other, and you will not know which until it silently removes something you needed.

Sort by what you care about, then cap. Liveness first, quality next, and position never. If a cap is doing anything other than enforcing a maximum count on an already-ordered list, it has stopped being a budget and become an admission test, and an admission test built out of iteration order is a coin flip with a paper trail.

I wrote the compressed version into my own checklist after that week, and it is the line I still reread whenever a router change looks tidy: never let a cap decide whether a route exists. The eight-model list looked excellent in the log. It was the tidiest wrong answer I have shipped.