Vercel just shipped virtual model IDs on AI Gateway, so you can change the provider, the fallback chain and the model behind a request centrally, without deploying a single line of application code. That's a great escape hatch. It's also a new way to change production behavior with no pull request, no diff and no test run. This post is the guardrail kit I'd wire up before the first person flips a route in the dashboard.
TL;DR
- Treat a Vercel AI Gateway virtual model ID as a contract: one stable name in your code, routing decisions owned somewhere else, and a written list of which errors may fall back.
- Log the model that actually answered on every call, using the OpenTelemetry GenAI attribute names, so a route change shows up in your traces the minute it happens.
- Keep a pinned concrete model next to the virtual one and run a tiny eval against both before and after every route change. That's your rollback and your proof.
Virtual model identifier
Here's the change in one sentence: your code asks for a stable name, and the gateway decides what that name means today. If you've read my guide to LLM model routing for cost and quality, this is the same idea moved out of your codebase and into the platform.
The first rule is boring and it matters most: the model ID lives in exactly one place. Not in five route handlers, not in a prompt file, not in a feature flag someone forgot about.
// src/ai/models.ts
// One source of truth for which model name the app asks for.
// AI_MODEL_ID holds the virtual model ID you created in the AI Gateway dashboard.
// AI_MODEL_PINNED holds a concrete provider/model ID you trust, used for evals and rollback.
function requireEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing env var ${name}`);
return value;
}
export const MODELS = {
chat: requireEnv("AI_MODEL_ID"),
pinned: requireEnv("AI_MODEL_PINNED"),
} as const;
export type ModelRole = keyof typeof MODELS;
Then every call goes through one function:
// src/ai/generate.ts
import { generateText } from "ai";
import { MODELS, type ModelRole } from "./models";
export async function ask(prompt: string, role: ModelRole = "chat") {
const requested = MODELS[role];
const result = await generateText({ model: requested, prompt });
// Log both sides: what you asked for and what the response says answered.
console.log(
JSON.stringify({
requested_model: requested,
response_model: result.response.modelId,
input_tokens: result.usage.inputTokens,
output_tokens: result.usage.outputTokens,
})
);
return result.text;
}
The AI SDK sends a plain string model ID through AI Gateway when AI_GATEWAY_API_KEY is set. Check in your own logs what response.modelId reports for a virtual ID before you build dashboards on it. I'd rather you see it with your own eyes than trust my guess about it.
Fallback contract and errors
Fallbacks sound like free reliability. They aren't. A fallback is a second model answering a question you tuned for the first one, so you need a contract that says when that's allowed.
Cloudflare AI Gateway documents the same pattern from the other side: a multistep fallback chain that moves to the next model after a request fails or times out, plus a cf-aig-step response header that tells you which step answered (step 0 is the primary, then 1, then 2). Whichever gateway you run, that's the shape to copy: a fallback happens for a reason you named, and the response tells you it happened.
My rule of thumb:
| Error | Fall back? | Why |
|---|---|---|
| Timeout | Yes | The provider is slow, your prompt is fine |
| 429 rate limit | Yes | Capacity problem, not a content problem |
| 5xx from the provider | Yes | Their outage, not your bug |
| 400 bad request | No, fail fast | A different model won't fix a malformed request |
| Policy refusal | No, fail fast and log | Silently retrying elsewhere hides a real decision |
If part of your stack still does its own retries (a second gateway, an older code path), make that rule explicit in code so the two layers don't fight:
// src/ai/fallback-policy.ts
export type FailureKind = "timeout" | "rate_limit" | "provider_down" | "bad_request" | "refused";
export function classify(status: number | undefined, err: unknown): FailureKind {
if (err instanceof DOMException && err.name === "TimeoutError") return "timeout";
if (status === 429) return "rate_limit";
if (status !== undefined && status >= 500) return "provider_down";
if (status === 400 || status === 422) return "bad_request";
return "refused";
}
const MAY_FALL_BACK: ReadonlySet<FailureKind> = new Set(["timeout", "rate_limit", "provider_down"]);
export function mayFallBack(kind: FailureKind): boolean {
return MAY_FALL_BACK.has(kind);
}
// Usage with a hard timeout on the request:
// const res = await fetch(url, { signal: AbortSignal.timeout(15_000), ... });
Pick your fallback models on purpose, too. A fallback should be a model you've already evaluated for that job, not whatever was cheapest that afternoon. My notes on choosing between Anthropic, OpenAI and Google models for agents are a decent starting list for that comparison.
Trace each model choice
Once routing lives in a dashboard, your traces are the only diff you get. OpenTelemetry publishes GenAI semantic conventions (version 1.44.0 at the time of writing) for exactly this: standard attribute names for model calls, so any backend can read them.
// src/ai/traced.ts
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { generateText } from "ai";
import { MODELS, type ModelRole } from "./models";
const tracer = trace.getTracer("ai-gateway-client");
export async function tracedAsk(prompt: string, role: ModelRole = "chat") {
const requested = MODELS[role];
return tracer.startActiveSpan(`chat ${requested}`, async (span) => {
span.setAttribute("gen_ai.operation.name", "chat");
span.setAttribute("gen_ai.request.model", requested);
try {
const result = await generateText({ model: requested, prompt });
span.setAttribute("gen_ai.response.model", result.response.modelId);
span.setAttribute("gen_ai.usage.input_tokens", result.usage.inputTokens ?? 0);
span.setAttribute("gen_ai.usage.output_tokens", result.usage.outputTokens ?? 0);
return result.text;
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
}
});
}
The query that matters: group by gen_ai.request.model and gen_ai.response.model. When someone edits a route, the pairs change, and you see it without anyone posting in Slack.
Roll back and evaluate outputs
Rolling back a virtual route is a dashboard action. Knowing whether you need to is the hard part. Keep a pinned model in your config and a small fixed eval set, and run both IDs side by side whenever a route changes:
// scripts/route-eval.ts (run with: npx tsx scripts/route-eval.ts)
import { generateText } from "ai";
const virtualId = process.env.AI_MODEL_ID!;
const pinnedId = process.env.AI_MODEL_PINNED!;
const cases = [
{ prompt: 'Return JSON {"intent": string} for: "cancel my order 4417"', mustInclude: ["intent"] },
{ prompt: 'Return JSON {"refund": boolean} for: "I want my money back"', mustInclude: ["refund"] },
];
function passes(text: string, keys: string[]): boolean {
try {
const obj = JSON.parse(text.replace(/```
json|
```/g, "").trim());
return keys.every((k) => k in obj);
} catch {
return false;
}
}
async function score(model: string) {
let ok = 0;
for (const c of cases) {
const { text } = await generateText({ model, prompt: c.prompt });
if (passes(text, c.mustInclude)) ok++;
}
return ok / cases.length;
}
const [v, p] = await Promise.all([score(virtualId), score(pinnedId)]);
console.log({ virtual: v, pinned: p });
if (v < p) {
console.error("Virtual route scores below the pinned model. Roll the route back.");
process.exit(1);
}
Two cases is a toy. Twenty real ones from your logs is a gate. Wire it into CI on a schedule, and run it by hand right after any route edit.
Here's my practitioner opinion: the teams that get burned by central routing aren't the ones without fallbacks. They're the ones who can't answer "which model answered this ticket last Tuesday?" Fix that first.
Cost and latency checklist
- [ ] One model ID constant in code, read from config
- [ ] A written list of errors that may fall back
- [ ]
gen_ai.response.modellogged on every call - [ ] A pinned concrete model kept next to the virtual one
- [ ] The route eval run before and after every change
- [ ] Cost per request and p95 latency reviewed per response model, every week
That last line is where routing pays for itself or doesn't. If you're still sizing what an agent feature will cost to run, the agentic AI MVP cost estimator gives you a first number to compare your traces against.
FAQ
How does Vercel AI Gateway virtual models work?
Your code asks for one stable virtual model ID. The gateway maps that ID to a provider, a model and a fallback chain that you manage centrally, so changing what answers the request doesn't need an application deploy.
What changes for teams adopting Vercel AI Gateway virtual models?
The change control moves. Model switches stop showing up as code diffs, so you need traces that record which model answered, a named list of errors that may fall back, and an eval you run around every route change.
Do I still need retries in my own code?
Only where something else calls providers directly. If the gateway owns fallbacks, keep your own layer to timeouts and logging so you don't retry twice.
Related reading: why model routing needs its own evaluation scorecard and how routing cut my inference bill without a quality drop.
How are you handling route changes today: dashboard only, or a gate in CI? Drop your setup in the comments, I'm curious what's working in production.




Top comments (0)