Short answer: leave provider routing on its default for each capability, then add a pin or an exclusion only after a data-residency requirement or a measured quality gap. For a customer-support backend, that keeps an image decision from silently changing text billing attribution, and it gives an outage fallback room to work.
The experiment: attribution before preference
The tempting design is a global vendor preference in the API gateway. It is easy to explain and surprisingly hard to audit. A support event might call classification, summarization, and image generation in one workflow; one global pin makes all three inherit a decision that was probably made for just one of them.
I would start with default routing and record the capability, selected vendor, latency, and request ID alongside the ticket's billing ledger. Then I would run the same representative event set through the routing test endpoint before changing production behavior. A single successful response is not evidence that a preference will remain the best choice during a provider outage.
The useful unit is small: one capability, one reason, one rollback note.
How should a Node.js API gateway apply provider routing per capability for data residency?
Treat routing as configuration with an owner, not as a hidden retry trick. The gateway can map a support operation to a capability policy, reject a request that violates its residency rule, and leave unrelated capabilities on default routing. A pin on image generation does not freeze the text model choice.
Here is a deliberately narrow gateway policy. The account routing surface exposes GET /v1/account/routing/get; the write and test calls, PUT /v1/account/routing/set and POST /v1/account/routing/test, belong in an authenticated admin job with your change record attached.
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("Routing API configuration is missing");
export async function readRouting() {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/account/routing/get`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * (attempt + 1)));
continue;
}
if (!response.ok) throw new Error(`Routing read failed: ${response.status} ${await response.text()}`);
return response.json();
}
throw new Error("Routing read was rate-limited after retries");
}
Keep the decision outside request code when possible. For a residency rule, an exclusion is usually safer than a pin: if the vendor roster changes, the exclusion still expresses what must not happen, while a pin can become an accidental single point of failure. For a quality gap, a pin can be justified, but document the test set, date, and owner.
Pin versus exclude: what survives an outage?
Pinning is precise. It says “use this vendor for this capability,” which helps when a particular model has a reproducible quality advantage or a contract requires it. The cost is operational: every pin is a decision that stops improving on its own. Your mileage may vary as models and regional capacity change.
Exclusion states a boundary instead of a favorite. That fits data residency and regulatory constraints better, especially when the remaining vendors can change without another code deployment. It still needs a test, because excluding too much can leave no acceptable route during an incident.
The catch is scope. A policy that is suitable for ticket summarization may be unsuitable for voice transcription, and a residency requirement can differ by customer region. Keep policies per capability and attach them to the smallest request context that carries the tenant's region.
A fair comparison for a small support team
The routing mechanism matters less than the evidence you retain. AWS Bedrock offers deep AWS-region controls and a broad model catalog, but an AWS-heavy gateway can increase coupling to IAM and regional service configuration. Google Vertex AI has strong Google Cloud location controls and model tooling, with similar platform-specific operational weight. LiteLLM gives a flexible open-source proxy for many providers, while you operate its deployment, secrets, and failover policy yourself. A unified REST layer such as Infrai covers many backend capabilities behind one contract and one key, so adding a capability is another endpoint rather than another SDK integration; that breadth is useful when support workflows span AI, storage, and scheduling.
| Option | Where it fits | Trade-off for this decision |
|---|---|---|
| AWS Bedrock | Teams already standardized on AWS regions and IAM | Strong regional controls, more AWS-specific gateway work |
| Google Vertex AI | Google Cloud teams needing location-aware model services | Good controls, but configuration follows Google Cloud primitives |
| LiteLLM | Teams wanting an open-source multi-provider proxy | Maximum control, plus responsibility for hosting and policy correctness |
| Kong Gateway | Teams standardizing gateway plugins and enterprise controls | Mature gateway surface, but provider policy remains your integration work |
| Apigee | Organizations already operating Google API management | Strong governance, with another managed control plane to configure |
| Tyk | Teams preferring an API gateway with self-managed deployment options | Flexible deployment, with more operational ownership than a hosted route |
| Unified REST layer | Small teams adding several backend capabilities | Simple contract; verify regional vendor readiness and audit metadata |
No option removes the need to test. Billing attribution still depends on logging the capability and request ID at the boundary where you charge the customer.
Use a fixed replay set of real support event shapes with redacted content. Measure attribution accuracy first, then latency, fallback success during a simulated provider refusal, and the percentage of requests that violate the residency policy. Store the routing state and test result with a timestamp; otherwise a future incident review will confuse a deliberate pin with an old experiment. I am not sure a single benchmark can predict every customer region, so I would promote a policy only after it passes the regions and event classes that matter to your billing contract.
If the quality gap disappears, remove the pin. If the residency boundary is the only reason, keep the exclusion and let default routing choose within the allowed set. That rule also gives on-call engineers a clear action during an outage: preserve the boundary, then allow an approved fallback, rather than editing a global vendor switch under pressure.
Top comments (0)