DEV Community

KiernanBerg3867
KiernanBerg3867

Posted on

Model Vendor Routing Constraints: A Prepaid API Balance Guard in Node.js

#ai #api #architecture #node

The constraint that decides this one isn't cost and it isn't benchmark quality. It's that the account funding the calls is prepaid, the batch runs at 03:00, and nobody is awake to approve a fallback. Our storefront regenerates product copy and triages return requests overnight, and both paths reach a model vendor through a single internal routing service that owns the API credentials.

Pick the allowlist.

Pin the vendor set, let the model id float inside that set, and treat "exclude vendor X" as a derived view rather than as the policy itself. An exclude list ages badly for one structural reason: it defines the allowed set as everything you haven't thought of yet. Every time the router gains a provider, the permitted surface grows on its own — new credential, new billing identity, new spend path that no one reviewed. An allowlist grows only when a human edits a file.

The overnight failure mode is an unreviewed route, not an overspend

The first version of this guard was an alert. Balance drops under 20%, webhook fires, someone tops it up. I assumed the alert was the control. It wasn't — an alert that lands in a channel at 03:14 is a notification, not a constraint, and the retry loop doesn't wait for a human to read it.

What made the incident annoying wasn't the money. A prepaid balance running out is a bounded loss; that's the whole point of prepaying. The problem was answering a question from the returns team two weeks later: which vendor's servers had seen the free-text return reasons from that night's batch? The router had failed over. The exclude list said "not vendor X," the job retried, and resolution landed on a provider whose key had been added months earlier for a one-off evaluation and never removed. Nothing was broken. The policy allowed it, because the policy was written as a subtraction.

Reconstructing that answer took longer than the outage. I had to rebuild the provider registry as it existed at 03:14, from deploy history and environment snapshots, because the effective allowed set was implicit. That's the cost you don't see in the design review.

Should I pin one model vendor or exclude one when the routing constraint has to stay auditable?

Pin, if the axis you actually care about is who can see the data and who can spend the balance. The exclude form is easier to write and easier to keep working; it fails on the forensics.

Here's the comparison that mattered to me, framed as the questions you get asked after something goes sideways rather than as feature bullets:

Question you get asked afterwards Vendor allowlist (pin) Vendor exclude list
Which vendors could have received this prompt? Read one file at one commit Reconstruct the router's full provider set at that timestamp
Who approved this spend path? The pull request that added the route Nobody explicitly; new providers inherit approval
What happens at zero balance? Hard stop, or one named standby Undefined — whatever ranks next
How do you revoke access? Delete the route, rotate one secret Add a rule and hope the enumeration is complete

The second column is boring on purpose. Pinning costs you flexibility, and if you pin down to a specific model id you also inherit that model's deprecation calendar, which is a real maintenance tax — vendors retire model ids on their own schedule and your pinned route stops resolving. So pin the vendor and the credential, and keep the model id a configurable string with a documented default. The vendor boundary is the one that maps onto a contract, a data processing agreement, and a bill. The model id doesn't.

One more thing about aging: policies age through the people who edit them, not through the code. An allowlist entry that nobody can explain gets deleted in review. An exclude rule that nobody can explain stays forever, because deleting it looks risky and nobody can prove what it was protecting against.

Routing as data, with the balance check on the dispatch path

The gateway holds about sixty lines of policy. Routes are data, checked into git, reviewed like a schema migration. Secrets are referenced by name and resolved at call time; the route table never holds key material, which is the baseline OWASP recommends and also what makes the table safe to log.

export type Route = {
  id: string;            // stable id, appears in every audit event
  vendor: string;        // contract boundary, not a model boundary
  baseUrl: string;
  secretRef: string;     // name of the secret, never the value
  defaultModel: string;
  monthlyCapUsd: number;
};

// The complete allowed set. Adding a line requires a reviewed commit.
export const ROUTES: Route[] = [
  {
    id: "catalog-copy",
    vendor: "vendor-a",
    baseUrl: "https://api.vendor-a.example/v1",
    secretRef: "LLM_KEY_VENDOR_A",
    defaultModel: "copy-large",
    monthlyCapUsd: 400,
  },
  {
    id: "returns-triage-standby",
    vendor: "vendor-b",
    baseUrl: "https://api.vendor-b.example/v1",
    secretRef: "LLM_KEY_VENDOR_B",
    defaultModel: "triage-medium",
    monthlyCapUsd: 150,
  },
];
Enter fullscreen mode Exit fullscreen mode

The dispatch function does three things before it opens a socket: resolve the route by id, refuse anything not in the table, and refuse to start a batch that the remaining balance can't finish. That last check is the one people skip, and it's the difference between a clean stop and a half-processed queue.

import { ROUTES, type Route } from "./routes.ts";

type Job = { routeId: string; estimatedUsd: number; batchId: string; itemHash: string };

export async function dispatch(job: Job, prompt: string, auditor: Auditor) {
  const route: Route | undefined = ROUTES.find((r) => r.id === job.routeId);
  if (!route) throw new Error(`route ${job.routeId} is not in the allowlist`);

  const balanceUsd = await readPrepaidBalance(route.vendor);
  // Reserve headroom for the whole batch, not for one call. A batch that
  // stops halfway leaves the queue in a state nobody wants to reconcile.
  if (balanceUsd < job.estimatedUsd * 1.2) {
    await auditor.emit({
      event: "route.blocked",
      reason: "insufficient_prepaid_balance",
      routeId: route.id,
      vendor: route.vendor,
      balanceUsd,
      requiredUsd: job.estimatedUsd,
      batchId: job.batchId,
    });
    throw new Error(`halted: ${route.vendor} balance ${balanceUsd} below reservation`);
  }

  const key = await secrets.read(route.secretRef);
  const started = Date.now();
  const res = await fetch(`${route.baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      authorization: `Bearer ${key}`,
      "content-type": "application/json",
      // Stable across retries, so a 429 backoff in the batch runner cannot
      // charge the prepaid balance twice for the same product description.
      "idempotency-key": `${job.batchId}:${job.itemHash}`,
    },
    body: JSON.stringify({ model: route.defaultModel, messages: [{ role: "user", content: prompt }] }),
  });

  await auditor.emit({
    event: "route.used",
    routeId: route.id,
    vendor: route.vendor,
    model: route.defaultModel,
    secretRef: route.secretRef,   // the name, never the key
    status: res.status,
    latencyMs: Date.now() - started,
    batchId: job.batchId,
  });

  return res;
}
Enter fullscreen mode Exit fullscreen mode

There's no fallback chain in there. That's deliberate. Failover is a separate, explicit decision with its own route id, and it never gets invented at runtime by a scoring function.

Retries live one layer up, in the batch runner: exponential backoff with jitter on 429, and an idempotency key derived from the batch id plus a hash of the item, so a retried write can't bill the prepaid balance twice for the same product description. A 402 is not retried at all — it's the signal to halt the batch and page, since retrying a payment-required response just burns the queue.

What the audit trail has to carry to be worth keeping

A log line that records "LLM call succeeded, 840 ms" answers nothing useful. The events above carry the route id, the vendor, the secret reference and the batch id, which is the minimum set for reconstructing access after the fact: who could have been called, which credential was used, and which unit of work it belonged to.

Log the name and version of the secret, never the secret. Rotation then becomes auditable too, because a mid-batch change in secretRef shows up as a visible seam in the trail instead of a silent substitution.

If you already run tracing, the OpenTelemetry generative-AI semantic conventions give you attribute names for the model and the request shape, which saves you from inventing a private schema that nobody else can query. I'd keep the spend and credential fields as your own attributes on top of that — the conventions cover the call, not your billing identity.

What to measure before copying this

Four numbers, all cheap to collect, and they tell you whether the allowlist is actually buying you anything.

Route entropy: how many distinct route ids served a single job type over 30 days. If it's one, your exclude list was never doing work and pinning costs you nothing. Credential drift: number of keys with nonzero spend versus number of routes in the policy — any gap means something is calling outside the table. Time from balance threshold to human acknowledgement, measured at 03:00 rather than at 14:00, because that's the number the design has to survive. And the cost of a hard stop, in delayed orders, which is the figure that tells you whether a standby route is worth its review overhead at all.

The catch is that this trade is wrong for a whole class of teams. If you're building an aggregator, or running weekly evaluations across a dozen models, an allowlist turns into a merge queue and your reviewers will start rubber-stamping route additions, which is worse than no policy because it looks like governance. Stick with an exclude list plus hard per-route spend caps there, and put your auditability budget into request-level provenance instead of into the routing table. Same if a latency SLO forces you into broad failover — you can't pin and also promise sub-second recovery across vendors.

And if you need reproducible outputs for a compliance artifact, pin the model id too, accept the deprecation churn, and accept that a retired model id means a hard stop. I'm not sure that trade is right outside regulated contexts; it depends on how much a stalled batch actually costs you, and for most storefronts it costs less than an unexplainable log.

References

Top comments (0)