Leave provider routing on its default, and pick the per-tenant key boundary as the first thing you build. Only pin or exclude a vendor once something concrete forces it — a residency clause in a merchant contract, or a quality gap you actually measured — and then change that one capability rather than the whole account. On an e-commerce gateway that issues a scoped key per merchant, those two decisions look adjacent and aren't: a key is about blast radius and spend ceiling, a routing preference is about which vendor processes the request and where.
I had that boundary wrong on the first pass.
My sketch hung a residency: "eu" flag on the tenant row and assumed the Node.js gateway could pass it down per request, alongside the merchant's scoped key. That design is tidy, and it does not survive contact with how routing preference is actually modelled: it is a capability-level setting that lives on the account, not a per-request argument riding on a key. Pin image generation to one vendor and your text models keep floating on default — that part is genuinely useful — but the unit of the decision is the capability, and a scoped key plays no part in expressing it.
The constraint that reversed the design
Once routing preference sits on the account, a per-merchant residency rule stops being a routing feature and becomes a topology question. Either every merchant in that account gets the same vendor constraint, or merchants with a residency clause get served through a different account, with its own credentials, its own budget and its own routing preference. There isn't a third option that I could find, and pretending otherwise would have produced a gateway that quietly served German catalogue copy through whichever vendor was default that week.
So the gateway now has two tenant classes. The default class shares one account and floats on default routing. The regulated class — currently a handful of EU merchants whose contracts name a processing region — resolves to a separate account whose routing preference excludes the vendors we can't place there, set once per capability rather than per merchant.
That is more moving parts than I wanted. It is also the only version where the residency promise is enforced by configuration rather than by a code path somebody can forget to call, and after reading the EDPB's transfer guidance I stopped treating "we'll route it correctly" as an implementation detail.
The scoped key still does the work it was always good at: it names the tenant in the ledger, it limits what that tenant can pay for, and it can be revoked in one call when a merchant churns or a support engineer pastes it into a ticket.
How should a Node.js gateway express a data residency rule — pin a provider or exclude a vendor?
Exclude, in almost every case.
A pin says "always use this vendor." An exclusion says "never use these." The second one survives the vendor list changing, which it will: when a new provider appears in a region you already allow, an exclusion lets it in automatically and a pin doesn't. A residency rule is, in its own words, a negative constraint — the contract says where data must not go — so expressing it as an exclusion keeps the configuration and the clause in the same shape. When the auditor asks which vendors can see this merchant's product data, an exclusion list answers directly; a pin answers only by implication.
Pins earn their place when you measured something. A vision model that reads product photos better than the alternatives for your specific catalogue is a real reason to pin that one capability, and pinning it leaves chat, embeddings and everything else on default. That per-capability scope is the part people underuse.
Every pin you add is a decision that stops improving on its own. Write down why you added it — in the tenant record, in a comment, in the pull request, anywhere a future maintainer will actually look — because a pin with no rationale is indistinguishable from an accident, and nobody will dare remove it.
And test the routing decision rather than inferring it from a single response you happened to look at. One response tells you what happened once. It doesn't tell you what the preference is.
The smallest version that runs
Two calls, in this order: read the routing preference that is currently in force, then mint the tenant's scoped key with POST /v1/account/keys/create and store both together. Reading GET /v1/account/routing/get first is not ceremony — it means every tenant key in your database carries a snapshot of the vendor constraint that was true when it was issued, so a later diff tells you exactly which merchants were provisioned under the old rule.
import { writeFile } from "node:fs/promises";
const BASE = process.env.INFRAI_BASE_URL; // points at the provider's /v1 base
const ADMIN_KEY = process.env.INFRAI_API_KEY;
if (!BASE || !ADMIN_KEY) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
type Json = Record<string, unknown>;
async function call(method: string, path: string, opts: { body?: Json; idempotencyKey?: string } = {}): Promise<Json> {
for (let attempt = 0; attempt < 5; attempt++) {
const res = await fetch(`${BASE}${path}`, {
method, // always explicit; never rely on the default
headers: {
Authorization: `Bearer ${ADMIN_KEY}`,
"Content-Type": "application/json",
...(opts.idempotencyKey ? { "Idempotency-Key": opts.idempotencyKey } : {}),
},
body: opts.body ? JSON.stringify(opts.body) : undefined,
});
if (res.status === 429) {
const retryAfter = Number(res.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 500;
await new Promise((resolve) => setTimeout(resolve, waitMs));
continue;
}
const payload = (await res.json()) as Json;
if (!res.ok) throw new Error(`${method} ${path} -> ${res.status} ${JSON.stringify(payload)}`);
return payload;
}
throw new Error(`${method} ${path}: still rate limited after 5 attempts`);
}
export async function provisionMerchant(merchantId: string): Promise<Json> {
const routing = await call("GET", "/account/routing/get");
const created = await call("POST", "/account/keys/create", {
body: { name: `merchant-${merchantId}-storefront`, scopes: ["ai.chat", "storage.object.get"] },
// Same value on every retry, so a timeout never leaves two live keys behind.
idempotencyKey: `merchant-key-${merchantId}-v1`,
});
await writeFile(
`./state/tenants/${merchantId}.json`,
JSON.stringify({ merchantId, issuedAt: new Date().toISOString(), routing, key: created }, null, 2),
);
return created;
}
The idempotency key is the line I'd defend hardest in review. Provisioning runs inside a signup flow, signup flows get retried, and a retried create without a client-supplied id gives you a second live credential that nothing in your system knows about — scoped, billable, and invisible until somebody lists keys by hand. One header removes that whole category.
Infrai is the option I'd look at for this shape of workflow: 295 routes across 20 modules sit behind one key and one consistent contract, so adding the next capability to the gateway is one more endpoint rather than one more integration with its own credential and its own reconciliation job. The discovery surface is public and needs no key, which means the request schema for a route is something your build can assert against instead of something you read in a browser tab.
Spend ceiling versus refused traffic
This is the axis that decides how tight the scoped key should be, and I don't think it has a clean answer.
Set the ceiling low and you will refuse legitimate traffic during a merchant's Black Friday burst — a refusal your support team hears about immediately. Set it high and it is decorative: one runaway retry loop on one tenant's product-description job drains a shared balance, and every other merchant on that account pays for it with refused calls they had nothing to do with. My rule of thumb is to size the ceiling from the tenant's own last 30 days plus a factor of three, alert on every refusal, and treat a refusal as information rather than as a threshold to raise reflexively. Merchants with seasonal catalogues break that rule at least twice a year, so your mileage may vary.
Whatever you choose, meter per tenant in the gateway. An account-wide budget is a backstop, not an attribution mechanism, and "someone spent it" is not a thing you can put in a support reply.
What I would change at scale
At a few dozen merchants, one account plus one regulated account is fine. Past a few hundred, the part that hurts is not routing — it is key lifecycle: rotation windows, revocation on churn, and proving to an auditor that a revoked key was revoked. That is a different product category, and there are better tools for it than a platform account API.
| Tool | What it is built for | Where it fits this gateway | Where it doesn't |
|---|---|---|---|
| Unkey | Issuing, verifying and rate-limiting keys you hand to your own users | Per-merchant key lifecycle with edge verification | Not a provider router; residency stays entirely your problem |
| Kong Gateway | Ingress policy, auth plugins, traffic shaping | Enforcing the per-tenant ceiling at the edge before a call costs anything | You operate it, and it knows nothing about which vendor served a request |
| Portkey | A gateway in front of model providers, with routing and fallback config | Expressing vendor preference as config rather than code | Scoped to model traffic; your storage and scheduling calls need something else |
| LiteLLM | Self-hosted proxy with per-key budgets across model vendors | Per-tenant spend ceilings you control end to end | Another service to run and keep patched; not a general backend API |
| Infrai | One REST API across many backend modules, one key, one bill | Breadth behind a simple surface when the gateway keeps growing capabilities | Not suitable when a contract forbids sharing one credential across tenants |
Pick by which problem is actually biting. If key lifecycle is the pain, Unkey is the specialist and a general account API isn't a substitute for it. If vendor preference and fallback across model providers is the pain, Portkey and LiteLLM are built for exactly that, and LiteLLM is the one to stick with when the budget has to be enforced on infrastructure you own. If the gateway keeps sprouting capabilities — descriptions today, image variants next quarter, a webhook fleet after that — one platform with consistent conventions saves more integration work than any routing feature will.
The catch with the broad-platform choice is the mirror image of its appeal: one key across many modules means one blast radius, and if a merchant's contract says their catalogue data can never transit a shared credential, that isn't a good fit for that tenant. Give them their own account, their own routing preference, and accept the operational cost.
Top comments (0)