Structured output holds until it does not, and the boundary is not where anyone guesses. Fuzzing here does not mean random bytes; it means growing one dimension of the schema at a time until the provider rejects the request or the model stops satisfying it, and recording the number.
Why a schema that works can stop working
Schemas grow. A field becomes an object, an enum picks up a hundred new values from a database, a shared definition gets referenced from three places and the nesting quietly doubles. Nothing about that process produces a moment where somebody asks whether the schema is still within the provider’s limits, so the first sign is a 400 in production on the one code path that uses the largest variant.
There are two distinct failures and they are worth separating from the start. A schema can be rejected, which is loud, immediate and easy to handle. Or it can be accepted and only partly honoured — the request succeeds, the output validates against the top level, and a deeply nested constraint or a long enum was not enforced. The second is the one fuzzing is really for, because nothing else in your stack will notice it.
The documented caps are the starting point
OpenAI publishes numeric limits for structured outputs, and its structured outputs guide states them plainly: at the time of writing, a schema may have up to 5000 object properties in total, with up to 10 levels of nesting; the total string length of all property names, definition names, enum values and const values cannot exceed 120,000 characters; a schema may have up to 1000 enum values across all enum properties; and where a single enum property has more than 250 values, the total string length of those values cannot exceed 15,000 characters. It also requires that every field be marked required and that objects set additionalProperties: false.
Those numbers give you the axes to fuzz and the values to bracket. They are also exactly the sort of figure that moves, and other providers claiming schema support publish different limits or none, so the point of the harness is to discover the limit rather than to assert the published one. Where the published number and your measured boundary disagree, the measured one is what your code has to live with.
These figures are what the provider documented at the time of writing and they have changed before. A test that hard-codes 10 as the nesting limit will fail spuriously the day it becomes 12; a test that reports the boundary it found will simply report a different number.
One axis per run
The temptation is to generate wild schemas with everything varying at once. Do not: when one fails you learn nothing, because you cannot say which property of it was the problem. Generate along one axis, hold everything else at a modest constant, and you get a boundary per axis that you can act on.
// gen.ts
type Schema = Record<string, any>;
const leaf = (): Schema => ({ type: "string" });
/** An object nested `depth` levels deep, one property per level. */
export function deepSchema(depth: number): Schema {
let node: Schema = { type: "object", properties: { value: leaf() },
required: ["value"], additionalProperties: false };
for (let i = 0; i < depth - 1; i++) {
node = { type: "object", properties: { child: node },
required: ["child"], additionalProperties: false };
}
return node;
}
/** A flat object with `n` string properties. */
export function wideSchema(n: number): Schema {
const properties: Schema = {};
for (let i = 0; i < n; i++) properties["field_" + i] = leaf();
return { type: "object", properties, required: Object.keys(properties),
additionalProperties: false };
}
/** One property whose enum has `n` members. */
export function enumSchema(n: number): Schema {
const values = Array.from({ length: n }, (_, i) => "value_" + i);
return { type: "object", properties: { choice: { type: "string", enum: values } },
required: ["choice"], additionalProperties: false };
}
Three generators, three numbers to sweep. Note that every generated object already sets additionalProperties: false and lists every property as required — otherwise the request is rejected for a reason that has nothing to do with the axis you are measuring, and your boundary comes back as 1.
The harness
Sweep by doubling to find the bracket and then bisect inside it, which costs a logarithmic number of requests rather than a linear one. Record three outcomes per point: accepted and honoured, accepted and not honoured, rejected.
// fuzz.ts
import Ajv from "ajv";
import { client, MODEL } from "./probe";
const ajv = new Ajv({ allErrors: true, strict: false });
export type Outcome = "honoured" | "unenforced" | "rejected";
export async function attempt(schema: object, prompt: string): Promise<Outcome> {
let content: string;
try {
const res = await client.chat.completions.create({
model: MODEL,
temperature: 0,
max_tokens: 2000,
response_format: {
type: "json_schema",
json_schema: { name: "probe", strict: true, schema },
},
messages: [{ role: "user", content: prompt }],
});
if (res.choices[0].finish_reason === "length") return "unenforced";
content = res.choices[0].message.content ?? "";
} catch {
return "rejected";
}
try {
return ajv.validate(schema, JSON.parse(content)) ? "honoured" : "unenforced";
} catch {
return "unenforced";
}
}
/** Largest n for which f(n) is "honoured", searched by doubling then bisecting. */
export async function boundary(
build: (n: number) => object,
prompt: string,
ceiling = 4096,
): Promise<number> {
let good = 1;
let bad = 2;
while (bad <= ceiling && (await attempt(build(bad), prompt)) === "honoured") {
good = bad;
bad *= 2;
}
while (bad - good > 1) {
const mid = Math.floor((good + bad) / 2);
if ((await attempt(build(mid), prompt)) === "honoured") good = mid;
else bad = mid;
}
return good;
}
The truncation check inside attempt is load-bearing. A wide schema demands a large output, so a run that sweeps width will eventually hit the token cap rather than a schema limit, and without that check you would report a schema boundary that is really a max_tokens boundary. Set the cap generously and treat length as a signal that the sweep has left the region it can measure.
Reading the result
The output is three numbers per provider and model — a depth, a width, an enum size — and their value is entirely in what you do with them. Turn each into a check that runs against your real schemas at build time: walk the schema, compute its depth, count its properties and its enum members, and fail the build if any exceeds the measured boundary with less than a comfortable margin. That check costs nothing per run and catches the field somebody adds next quarter.
Expect the boundaries to differ from the published caps in one direction more than the other. A schema well inside the documented limits can still come back unenforced, because enforcement and acceptance are separate mechanisms and the model can be given a grammar it satisfies only loosely at depth. That result is more useful than a rejection: it tells you which parts of your schema you must still validate yourself after parsing, which is the honest answer for any nested structure regardless of what the provider promises.
Two practical notes about running this at all. It costs real requests — the doubling-and-bisecting search is perhaps twenty per axis, and a wide schema means a large completion, so this is the most expensive suite in the cluster by a wide margin. Run it when you adopt a provider, when you adopt a model, and otherwise on a quarterly schedule, never on a commit hook. And give it a cheap prompt: the content of the response is irrelevant to every assertion here, so ask for something the model can answer in one pass rather than something that makes it think.
Finally, keep the recorded boundaries in a committed file with the date and the model id, the same way the capability matrix is kept. A boundary without a model id is not a fact about anything — two models from the same vendor can differ here, and a number that quietly refers to a model you stopped using is worse than no number, because a build-time check will happily enforce it.
Top comments (0)