We're building TokenBay, a unified API that routes requests across GPT, Claude, Gemini, DeepSeek, and a few dozen other models. A few weeks ago we ran a fairly boring internal test: send the same structured-output prompt to five different models and compare what came back. It turned into one of the more useful debugging sessions we've had this quarter, mostly because nothing about it looked broken at first.
The task was simple. Extract product, budget, and location from a user request, return JSON. Every request succeeded. Every model returned something a human would call correct. And our downstream workflow — the thing that actually parses the response and does something with it — broke anyway.
The responses all looked fine
Here's the input we sent:
I need a laptop under $800. I'm in California.
GPT-5.4 came back with exactly what we expected:
{
"product": "laptop",
"budget": 800,
"location": "California"
}
Claude Opus 4.8 gave us the right values, but wrapped in a sentence first:
`Here is the extracted information:
{
"product": "laptop",
"budget": "$800",
"location": "California"
}`
Still correct, if you're a human reading it. Gemini 3.1 changed the field names entirely:
{
"item": "laptop",
"max_price": 800,
"region": "California"
}
Also correct. DeepSeek V4 followed the schema but represented the missing currency differently again. Four models, four slightly different shapes, and our parser — which was expecting exactly product, budget, and location — choked on three out of four.
A unified endpoint doesn't mean unified behavior
This is the part that's easy to underestimate when you're evaluating a multi-model API. The pitch is usually about the request layer: one key, one endpoint, swap the model name and you're done. That part is true, and it's genuinely useful. But it quietly implies the models are interchangeable once you can call them the same way, and they aren't.
They interpret the same instruction differently. They make different assumptions about missing values. They disagree on how strict "return JSON" actually means. Two models can agree completely on the answer and disagree completely on how that answer should be formatted — and that difference is invisible in a demo, because a person is reading the output. In production, the next thing reading it is usually another piece of software, and it's a lot less forgiving than a person skimming a chat window.
Tightening the prompt helped, but not enough
Our first move was the obvious one — make the instruction stricter. We went from:
Return the result as JSON.
to
Return only valid JSON. Do not include any explanation or Markdown.
That fixed the Claude wrapper-text problem immediately. It didn't fix everything. Some models represented a missing field as null, some as an empty string, some just omitted the key entirely:
{ "product": "laptop", "budget": 800, "location": null }
{ "product": "laptop", "budget": 800, "location": "" }
{ "product": "laptop", "budget": 800 }
All three are valid JSON. None of them are the same contract. At that point it was clear we weren't asking the models for a format — we needed to ask our own workflow to enforce a schema, because the models were never going to agree on one for us.
The actual fix was a normalization layer, not a smarter prompt
Prompt tuning bought us a few percentage points, but it's fragile by nature — a model can follow an instruction 99 times out of 100 and still surprise you on the hundredth call. If that hundredth response ends up in a billing flow or an automated action, "usually correct" isn't a bar we were willing to accept.
So we stopped passing raw model output directly into the app. Everything now goes through a small adapter first: pull JSON out of any surrounding text, map alternate field names onto our internal schema, coerce currency strings into numbers, collapse empty strings to null, and reject anything incomplete outright. A simplified version of what that looks like:
type RawResult = Record<string, unknown>;
type ProductRequest = {
product: string;
budget: number | null;
location: string | null;
};
function normalizeResult(raw: RawResult): ProductRequest {
const product = raw.product ?? raw.item;
const rawBudget = raw.budget ?? raw.max_price;
const location = raw.location ?? raw.region ?? null;
if (typeof product !== "string" || product.trim() === "") {
throw new Error("Missing product");
}
let budget: number | null = null;
if (typeof rawBudget === "number") {
budget = rawBudget;
} else if (typeof rawBudget === "string") {
const parsed = Number(rawBudget.replace(/[^0-9.]/g, ""));
if (!Number.isNaN(parsed)) {
budget = parsed;
}
}
return {
product: product.trim(),
budget,
location:
typeof location === "string" && location.trim() !== ""
? location.trim()
: null,
};
}
Nothing here is clever, and that's kind of the point — the real production version handles a longer list of edge cases (arrays instead of strings, nested objects, models that helpfully "correct" your currency to EUR), but the core idea doesn't change: stop trusting the raw shape and normalize before anything downstream touches it.
This also changed what we count as a failure
Before this, our fallback logic only triggered on request failure — timeout, rate limit, a 4xx or 5xx from the provider. If a request came back as 200 OK, we treated it as done.
But a 200 with an unusable payload isn't a success, it's a failure wearing a success status code. So we moved the fallback trigger downstream:
Send request → Receive response → Parse output → Validate schema → Normalize → Continue, or retry with another model
The question changed from "did the API respond?" to "did the API produce something the application can actually use?" Those aren't the same question, and building fallback logic around the first one instead of the second one will pass every test you write until it doesn't.
What we test now
We used to eyeball a handful of outputs and call the prompt "good enough." Now the test suite specifically checks for the boring stuff: extra text wrapped around the JSON, renamed fields, numbers returned as strings, omitted optional fields, inconsistent handling of missing values, and whether the normalized result from every model passes the exact same downstream checks. A response can be fluent, accurate, and still completely unusable to a parser — that's its own failure mode, and it needs its own test.
The takeaway
Being able to call five models through one API makes them easier to reach. It doesn't make them interchangeable. The call can succeed, the model can understand the task perfectly, the answer can even be right — and the workflow can still break quietly downstream. Routing the request to a different model is a config change. Deciding what the rest of your application is allowed to assume about that model's output is the actual engineering work, and it's the part that doesn't show up in a benchmark chart.
We ran into this while building TokenBay, a unified API for routing across GPT, Claude, Gemini, DeepSeek and other models. If you've hit similar cross-model quirks, I'd be curious to hear how you handled them.
Top comments (0)