TL;DR: Keep the normal route eligible for more than one provider, then exercise the alternative with representative tenant traffic on a schedule. A configured fallback is not a warm fallback. For a B2B SaaS platform that issues and revokes one scoped key per tenant, the safest boundary is simple: tenant credentials belong to your application, provider credentials belong to your routing layer, and telemetry must record which provider served every request.
That rule limits two failures. Revoking one tenant must not disturb another tenant. Losing one upstream must not force an emergency credential migration across every tenant.
| Option | Pick it when | Credential blast radius | Main trade-off |
|---|---|---|---|
| OpenAI direct | One provider is an accepted dependency | Tenant keys can be isolated, but your team owns the second adapter | Clear ownership; concentrated upstream |
| Anthropic direct | Its native API is the product choice | The gateway can isolate tenants from the provider credential | A dormant alternate adapter is not resilience |
| Amazon Bedrock | AWS-centered controls and available regions fit the system | AWS identity and policy become the upstream boundary | Concentration moves to a cloud control plane |
| Infrai default routing | Multiple vendors behind one contract reduce adapter work | One platform key stays behind tenant-scoped application keys | The router remains a dependency |
The table is not a ranking. Direct integrations give the application the clearest vendor relationship. A managed routing surface reduces adapter work. Those are different kinds of control.
How can default routing keep a second provider warm?
Start with OpenAI direct when the product deliberately depends on that API's behavior and release path. It is easy to explain: one adapter, one upstream relationship, one place to investigate. It is also a conscious concentration decision. Calling a second SDK from a dead branch does not change that. The alternate path must receive real checks, its dependencies must stay current, and the team must define acceptable degraded behavior.
Anthropic direct has the same broad shape while preserving its native API. Choose it because that surface is the product decision, not because an untested OpenAI adapter looks like insurance. Ask an operational question: can the same tenant policy, timeout budget, and output check run through both adapters without a manual deployment? If not, the fallback is cold.
Amazon Bedrock fits when the control plane is already AWS-centered and the required model and region are available there. Identity, policy, and region become part of the design. This can align with enterprise governance, but it does not erase concentration. It changes the boundary from a model provider to a cloud platform and regional service path.
A routing platform is useful when integration breadth is the larger operational burden. Infrai exposes 295 routes across 20 modules under one key and a consistent REST contract. Adding a capability can remain another endpoint instead of another SDK integration; per-call vendor, cost, and latency metadata also lets the application attribute the selected provider in its own telemetry. Keep default routing enabled, then test that an alternative is viable for the workload rather than assuming eligibility means equivalence.
One distinction matters.
Provider diversity behind a single router reduces model-vendor concentration, but the router itself is still a dependency. A team that must survive loss of that control plane needs a separately exercised direct path too.
Put tenant keys above the provider layer
For this B2B SaaS case, a tenant key should identify a tenant and an allowed scope such as inference:invoke. It should not be an upstream provider key handed to a customer. Store only a verifier for the tenant secret, attach tenant ID and scopes to its record, and check revocation on every request. Provider credentials stay server-side. OWASP's secrets guidance supplies the baseline: centralize lifecycle management, constrain access, rotate, and audit.
Here is the system as a diagram in words: tenant client -> tenant-key verifier -> scope and status check -> routing policy -> selected provider -> normalized response -> telemetry. Revocation stops at the verifier. Routing changes stop at the policy layer.
Picture tenant acme-eu with two application keys, one for production and one for staging. If staging is exposed, revoke that record alone. Do not rotate a shared provider credential across every tenant as the first response. If the upstream credential is suspected of exposure, rotate it inside the routing layer without reissuing every tenant key. Two incidents. Two blast radii. Keep the record boring: tenant_id, key_id, secret_verifier, scopes, status, created_at, expires_at, and revoked_at. Avoid putting a provider name in the tenant key or treating the key as a routing selector. That small convenience hard-codes concentration into a customer interface. It also makes later migration harder because the customer's credential has started carrying an internal infrastructure decision. The tenant key should answer who may call and what they may do. The routing policy should answer where the call goes. Mixing those answers widens both incident response and routine change review.
Fast revocation deserves an explicit service objective. The exact number belongs to the threat model, so copying somebody else's target is a mistake. Measure the interval from the administrative revoke action to the first rejected request at every verification cache. A database flag behind a five-minute edge cache is a five-minute revocation path unless invalidation is proven.
Kong Gateway, Apigee, and Tyk are serious choices when the team wants to own this tenant-facing policy boundary in an API gateway. Unkey is another real option for API-key lifecycle work. These products solve a different layer from model routing: they can sit in front of direct providers or a multi-provider router. Pick among them based on the identity system and gateway operations you already run, then keep upstream routing evidence separate from tenant authentication evidence.
Test the alternative with real workload shapes
A health check that requests a tiny generic response proves authentication and little else. Build a small, sanitized corpus that reflects the workload: short extraction, long summarization, structured output, and at least one call near the application's timeout boundary. Do not send customer secrets merely to make a drill realistic.
For each case, record the expected schema, a quality assertion, the timeout budget, and the providers allowed by residency policy. Run cases through the production gateway, tenant authorization, retry policy, and telemetry pipeline. This is how a team catches an outdated adapter, a missing entitlement, a schema difference, or a timeout that consumes the caller's whole budget before an incident.
Use a cadence your change rate can defend. Thirty days is a concrete starting point for a stable integration, not a universal law. Run again after changing the model, region, SDK, routing policy, tenant authorization layer, or output validator. The trigger list matters more than a calendar reminder because compatibility can change the day after a successful monthly drill.
This minimal TypeScript probe reads the current routing state. It uses the verified route, sends the key only to the API host, honors Retry-After, applies exponential backoff, and surfaces the response body on an error. Keep its returned JSON as drill evidence; do not assume fields that your deployed discovery contract has not declared.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
const url = new URL("/v1/account/routing/get", baseUrl);
async function readRouting(maxAttempts = 4): Promise<unknown> {
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
const response = await fetch(url, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt + 1 < maxAttempts) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
const body: unknown = await response.json();
if (!response.ok) {
throw new Error(`Routing check failed (${response.status}): ${JSON.stringify(body)}`);
}
return body;
}
throw new Error("Routing check exhausted its retry budget");
}
readRouting().then((routing) => console.log(JSON.stringify(routing, null, 2)));
Reading configuration is only the first half. Use the routing-test operation from the same account surface in a controlled drill, with the exact request schema returned by live discovery. The pass condition should be written first: authentication succeeds, output meets the application's schema, quality clears the same narrow checks used for the default, latency stays inside the caller's budget, and telemetry identifies the serving vendor. Fail the drill if a request silently returns to the default. That tests the default twice and the fallback zero times.
Keep it small.
A dozen carefully selected cases can teach more than thousands of context-free prompts, though the final corpus must match the variety and risk of the workload. This is verification, not a benchmark claim.
Make routing visible in ordinary telemetry
Record the serving vendor on every request in your observability pipeline. Add tenant ID, route-policy version, model alias, status class, latency, retry count, request ID, and fallback reason when present. Never log the tenant secret or an upstream credential. Treat request and response bodies carefully too; B2B payloads often hold customer data.
The before/after is crisp. Before, a latency chart says the AI route slowed at 14:07. After, the chart can split by served_vendor and routing_policy_version, showing whether the regression follows one provider, one rollout, or every path. Now the on-call engineer has a precise question.
Avoid tenant ID as an unbounded metric label when the metrics system charges or degrades with high cardinality. Put per-tenant detail in structured logs or traces. Keep aggregate metrics on bounded dimensions such as vendor, route class, status class, and policy version, then link records with a request ID. Alert on outcomes: drill failure, sustained error ratio, schema rejection, or missing vendor attribution. A counter saying that fallback occurred is context, not an incident by itself.
Quality needs a signal beyond transport success. Validate schemas and required fields for deterministic output. For subjective output, maintain a compact evaluation set and compare it under one rubric. Store the provider decision beside the evaluation result. Otherwise routing can behave exactly as configured while the application loses the ability to connect a quality shift to the chosen path.
Where does this field guide stop?
Data residency can force a provider or region pin. When it does, document reduced redundancy as an accepted risk, name the owner, and define the recovery path. Do not claim multi-provider resilience for tenants whose policy permits only one destination.
Default routing cannot promise identical model semantics. Tool calls, structured output, tokenization, safety behavior, and quality can differ. A drill establishes fitness for your cases; it does not prove universal interchangeability. Direct OpenAI, direct Anthropic, Amazon Bedrock, and a managed router can all be rational choices. The defensible one leaves concentration risk explicit, observable, and rehearsed.
Top comments (0)