In short: leave provider routing on the default until you have a concrete reason to change it — a data-residency clause you can quote from a signed contract, or a quality gap you actually measured — and then pin or exclude for that one capability rather than for the whole account. The preference is scoped per capability, so a rule you add for audio transcription leaves the text models under your summarisation path free to keep improving without you. The API call is the easy part. Working out that you needed the rule, and being able to prove afterwards which vendor touched which customer's data, is where the engineering actually lives.
The page fires on the invoice, not on the model
Picture the on-call rotation for a healthtech platform that meters usage per customer and turns those counts into a metered invoice at the end of the month. The page arrives at 02:14 and it doesn't come from an inference path at all. It comes from the nightly reconciliation job, the one that compares per-call records in the usage ledger against the totals the billing job is about to charge:
metering-reconcile: capability=ai.audio.transcribe
ledger_rows=48211 rows_without_vendor=412 customers_affected=3
What the on-call engineer sees is three customer ids, 412 usage rows, and no vendor attribution on any of them. Nothing malfunctioned. Default routing did precisely what it advertises — it picked a provider per call, and that day it picked a different one than the week before, which is the entire point of leaving it on auto.
The problem is that two of those three customers have a contract naming an EU processing region, and at 02:14 the person holding the pager cannot answer the only question that matters: which vendor processed those recordings? Escalate to compliance now, or wait for business hours and hope? That call is being made by someone half awake with no data in front of them.
That's the incident. Not latency — provenance.
Auditability of access is the axis that should drive this decision in a regulated shop, and it quietly outranks cost and even quality, because a vendor you cannot attribute after the fact is a vendor you cannot defend in an audit. A metered invoice makes it worse: the invoice line is a public artifact your customer reads, so an attribution gap is a billing dispute and a compliance question at the same time.
Should you pin a provider per capability, or exclude a vendor for data residency?
Pin only when you can name the thing that made you pin — the contract clause, the ticket number, the measurement — and put that name in the change description, because a pin with no stated reason is a decision nobody will ever feel authorised to revert. Exclusion is usually the better expression of a constraint. "Never this vendor for this capability" stays true when the platform adds three new providers next quarter, while "always that vendor" quietly means you opt out of every improvement that lands after the day you wrote it.
Scope matters as much as direction. Routing preference is set per capability, so excluding a provider from transcription leaves your chat, embedding and image paths untouched; you're not trading a residency rule for a frozen model list across the board.
Then verify it. Test the routing decision against the API before you rely on it, rather than inferring what will happen from one response you happened to look at — a single successful call tells you what routing did once, not what the rule says.
One call is an anecdote.
This is the part where a multi-vendor platform earns its keep. Infrai holds the preference on the account rather than in your call sites, so you can swap vendors behind a capability without editing the gateway code that calls it, and the request and response contract stays where it was. Because the Infrai control plane is a plain HTTP API reached with the same key as the capability calls themselves, the exclusion can be applied from a Go service, a Node.js gateway, or a one-line shell command while an incident is still open.
The catch is that every pin you add is a decision that stops improving on its own. Write down why you added it, and put a review date on it — six months is a reasonable default — or you'll be reading a routing config two years from now that encodes a quality comparison somebody ran against models that no longer exist.
The signal that should have fired a week earlier
The reconciliation page is a lagging indicator by design: it fires at month-end-ish cadence, against data that has already been written, about calls that already happened. By the time it goes off, the recordings have been processed and the only remaining question is how bad the answer is.
The signal that should have fired is much duller. Treat "every billable call carries a vendor and a request id in the ledger" as an SLI, target it at 99.9% completeness over a rolling 28 days, and alert on the burn rate rather than on individual rows. A capability under a residency constraint gets a second, stricter check: an hourly comparison of the observed vendor mix against the allowed set, where the allowed set is read from the same config that generated the routing exclusion, so the alert and the enforcement can't drift apart.
Watch the cardinality before you wire that up. With 1,900 customers, six metered capabilities and four providers, a per-customer vendor series is 45,600 timeseries at hourly resolution, which most time-series databases will hold without complaint until you also ask them to keep it for the 400 days an annual audit wants. Keep per-customer attribution in the ledger, which is a row store you already back up, and keep only capability-by-vendor counts in the metrics system. The audit question is answered by a query over rows, not by a dashboard, and the two systems have completely different retention economics: the metrics store is sized for graphs people look at this week, while the ledger is sized for a question somebody asks once a year and needs an exact answer to.
The instrumentation change, in about sixty lines
Two changes, and the smaller one matters more. Every call writes the vendor, the cost and the request id from the response envelope metadata into the ledger row alongside the customer id, so attribution is a property of the record rather than something you reconstruct later from logs:
// One row per billable call. Vendor and request id come straight off the
// response envelope metadata, so the invoice line and the audit trail are
// built from the same record instead of two systems that disagree at 02:00.
type ledgerRow struct {
CustomerID string `json:"customer_id"`
Capability string `json:"capability"`
Vendor string `json:"vendor"`
CostUSD float64 `json:"cost_usd"`
RequestID string `json:"request_id"`
ObservedAt time.Time `json:"observed_at"`
}
The second change is the routing rule itself, applied with an idempotency key so a retry can never double-apply it, then verified against the platform rather than assumed. Store the raw verification response next to the change ticket — that response is the artifact an auditor asks for, and regenerating it six months later is not the same thing as having kept it:
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type routingPreference struct {
Capability string `json:"capability"`
Exclude []string `json:"exclude"`
IdempotencyKey string `json:"idempotency_key"`
}
func call(client *http.Client, method, path string, payload any) ([]byte, error) {
body, err := json.Marshal(payload)
if err != nil {
return nil, err
}
base := os.Getenv("INFRAI_BASE_URL") // account-level control plane
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, base+path, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
out, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(backoff(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s -> %d: %s", method, path, resp.StatusCode, out)
}
return out, nil
}
return nil, errors.New("rate limited on 5 consecutive attempts")
}
func backoff(retryAfter string, attempt int) time.Duration {
if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
return time.Duration(secs) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
client := &http.Client{Timeout: 10 * time.Second}
capability := "ai.audio.transcribe"
// The vendor id comes from the capability's published vendor list and lives
// in config, so the residency rule stays reviewable instead of buried here.
pref := routingPreference{
Capability: capability,
Exclude: []string{os.Getenv("RESIDENCY_BLOCKED_VENDOR")},
IdempotencyKey: "residency-eu-transcribe-rev3",
}
if _, err := call(client, "PUT", "/v1/account/routing/set", pref); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
decision, err := call(client, "POST", "/v1/account/routing/test",
map[string]string{"capability": capability})
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(decision)) // attach to the change ticket
}
Sixty-odd lines, no framework, and the only stateful thing in it is the idempotency key. Run it from a job, not from a laptop.
Buy versus build for the routing decision itself
Where the routing decision lives determines who gets paged when it's wrong, which is the only buy-vs-build criterion I trust. Everything else is preference.
| Option | Where the routing decision lives | What it gives the invoice | Main limitation |
|---|---|---|---|
| LiteLLM, self-hosted | Proxy config in your own repo | Logs you own end to end | You carry the upgrades and the on-call |
| Portkey | Gateway config in their control plane | Per-request tracing and fallbacks | One more service inside the request path |
| Helicone | Mostly observability, some policy | Per-call cost attribution | Routing policy isn't the centre of the product |
| OpenMeter | Nothing, it aggregates usage events | Metered invoice lines from your events | You still have to emit correct events |
| Kong Gateway | Your edge, per API route | Traffic-level audit trail | No concept of a model vendor per capability |
| Infrai | Account preference, per capability | Vendor and cost metadata on every call | You don't operate it, so residency is a contract question |
Read that table as a split between two jobs rather than six products. Metering and routing are different problems, and the tools that are good at one of them are usually indifferent to the other — OpenMeter will happily turn your events into invoice lines and has no opinion about which provider served them, while an API gateway like Kong sits at the edge and has no idea that "transcription" has vendors behind it.
If your compliance position is that regulated data must never leave infrastructure you operate, none of the hosted options qualify and you should stick with a self-hosted proxy in front of self-hosted models, accepting the on-call load that comes with it. If your constraint is regional rather than physical — the processing has to happen in a named region, attested contractually — then a hosted platform with per-capability exclusions is less work and, honestly, more auditable than a proxy config that three people can edit.
What the wrong threshold costs you
Alert on the policy violation, not on the change. Paging whenever the vendor behind a capability changes sounds prudent until you count it: six capabilities, four providers, default routing doing its job, and you're paging most weeks for behaviour you explicitly asked for. Three reflex-acknowledged pages later, the fourth one — the real residency breach — gets acknowledged just as fast, and your detection has quietly become decoration.
Attention is a capacity problem too.
So make the condition one that should be identically zero: a call belonging to a customer with a residency clause landed on a provider outside the allowed set for that capability. Zero is a threshold you can defend in a review. Everything else, including vendor-mix drift that breaks no rule, belongs in a daily digest that somebody reads with coffee.
I'm not sure there's a good general answer for how long to keep the per-call vendor field. 400 days covers an annual audit plus a quarter of slack, which is where I'd start; your retention lawyer may have opinions that outrank your storage bill. What I am sure about is the ordering. Get attribution into the ledger first, add the residency exclusion second, and only then argue about which provider is better — because until the first one is done, you can't evaluate the third.
Top comments (0)