The alarm I remember best did not come from our monitoring stack. It came from the billing page, and it described a batch job that, as far as our dashboards were concerned, was still running perfectly. Nothing in our latency charts moved, and none of our error budgets burned, yet the cost graph climbed at a rate that made little sense for the model we believed we were calling.
That belief was the problem. We had built a small inference router to send routine classification work to a lighter model, and we let the router decide what to call. For a few weeks the plan worked so well that we stopped looking at the routing table at all. Then a routine dependency update changed which model names resolved to which endpoints, and the router happily sent thousands of batch items to a much more expensive target because, in its view, the target was still healthy.
This article is the postmortem I ended up writing for that failure. It is not a product review and not a benchmark, because the real lesson was not about which model wins. It was about the fact that a router with no visibility into cost is just a load balancer that makes expensive mistakes politely.
What happened, in order
The timeline starts at 02:47 when the batch queue filled faster than usual. At 06:00 the queue had drained, and the latency numbers looked like any other morning. At 09:30 someone opened the billing dashboard and noticed the day's spend was closer to a week's budget than a day's. At 11:00 we confirmed that the routing table contained a mapping we had not written, and by 14:00 we had reproduced the entire sequence on a single test account.
The contributing factors were less dramatic. The routing table sorted candidates by estimated cost, so cheaper models appeared first. The health checks verified that an endpoint answered, but they did not verify which model answered behind it. The queue manager treated a slow response as a normal condition and simply retried more aggressively. Finally, nobody had a test that asserted the caller of a given batch was still the model we thought it was.
Any one of those conditions alone would have been harmless. All four together produced a slow, silent drift: the cheap lane stayed green in the control panel while the expensive lane answered the traffic, and the queued work followed the answer.
The fix that actually held
We replaced the cost-sorted table with a rule that belongs to the batch job itself, not to the router. Each batch card now declares an expected model group, and the router must prove that the endpoint still reports that group before traffic moves. If the endpoint's identity does not match, the job fails fast instead of silently merging into the expensive lane.
We also added a cheap-to-run verification step that runs before every batch cycle. It sends one tiny request to the cheapest configured path, captures a marker from the response, and compares that marker with the one stored alongside the model's configuration. You can implement this pattern in roughly fifty lines, and it is the only reason the same accident has not happened again.
Here is the minimal check that would have caught our failure, reduced to a script you can adapt:
import yaml
import requests
with open("routes.yaml") as f:
routes = yaml.safe_load(f)
expected_marker = routes["cheap_lane"]["marker"]
probe = requests.get(routes["cheap_lane"]["health_url"], timeout=5)
if probe.status_code != 200:
print("cheap lane down; paging human")
raise SystemExit(1)
actual_marker = extract_marker(probe.text)
if actual_marker != expected_marker:
print("cheap lane now answers as another model; refusing batch")
raise SystemExit(1)
print("cheap lane still cheap; proceeding")
The script is deliberately simple because the point is not sophistication. The point is to make the routing decision observable on every run, so a silent upstream change turns into a loud, early failure instead of a billing surprise.
Where free endpoints fit
This is also where a no-key development path changes your ability to test such a system honestly. If your experiment requires a payment method before you can even see the routing behavior, you will probably skip the experiment, or you will run it once and then let the test become stale. When you can point a router at a free model endpoint and run the same probe regularly, the verification step becomes something you can leave enabled forever instead of an activity that costs money.
MonkeyCode keeps offering exactly that combination right now: free model access for prototype traffic and a free server option where you can host such a probe. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention the project here because it let me run the probe from above for weeks without touching a payment form, and the free server meant the cron job did not live on my laptop. That is a comfortable way to keep a cost-safety check alive, and it matched the workflow I would recommend anyway.
Who should not copy this approach
If your batch jobs are small enough that a wrong model costs you a rounding error, the probe adds operational weight without much benefit. If your routing table is managed by a vendor that already provides usage groups, you should use that native signal first. And if you handle regulated data that cannot leave your network, the free server option is obviously not for you; run the identical script against an internal allowed host and keep the same check there.
What I would keep from this incident, regardless of provider, is the habit of asserting identity before trusting a route. Cost is not a property of the client. Cost is a property of whatever ends up answering the request, and any caching layer, alias, or dependency update can change that answer while leaving your dashboards green.
If you take one thing from this writeup, make it the fifty-line probe. Schedule it, watch it fail at least once, and then enjoy the boring months where it never has to say anything at all.
Top comments (0)