A schema bug, a retry that couldn't help, and a small design rule I'm keeping.
The error
One day a StructFlow job — the structured-extraction service on my document-processing API — failed with an error I had never seen:
Schemas contains too many parameters with union types (27 parameters with
type arrays or anyOf). This causes exponential compilation cost. Reduce the
number of nullable or union-typed parameters (limit: 16 parameters with unions).
My first guess was length — the schema must be too big. Wrong. The input wasn't unusually long. More to the point, I had never written this schema at all: on this API, the output schema is derived from each user's example_output — a JSON example of the shape they want. That detail matters. A fixed internal schema either fits under a limit forever or fails on day one; my services built that way had run quietly for months. This error could only come from the open-ended part of the system — whatever example a user decides to send. And even in that user's example, nobody had written 27 anywhere.
Where 27 comes from
Two mechanisms multiply, and neither is visible on its own.
First, the strict-conversion rules. The shared schema generator derives required as the intersection of keys present across all example items; keys that don't appear in every item are simply left out of required. No unions yet — this intermediate schema is byte-identical for every provider. The unions are manufactured one step later, by the provider-specific strict conversion. Providers whose strict contract demands that every property be listed in required leave the converter only one way to express "optional": a nullable union —
"id": { "type": ["string", "null"] }
So a heterogeneous example — sections that are sometimes tables, sometimes procedures, sometimes notices, sometimes figures — quietly produces nine union-typed parameters in one schema. Nobody typed the word "union": not the user who wrote the example, not me. The provider's contract forced my converter to manufacture them, following rules I wrote once and then stopped thinking about.
Second, batch composition. To use tokens efficiently, the layer above packs multiple input units into one request, and the response schema is repeated per unit (index0, index1, ...). Nine unions per unit × three units = 27.
The constraint isn't about schema length. It's per-unit unions × batch size — a multiplication where I controlled both factors separately and had never once looked at the product.
The retry that couldn't help
Here is the embarrassing part. The error did trigger my retry logic.
My pipeline is built on a principle I've kept for 24 years: absorb failures with deliberate retries and always return a result to the user. The retries fired exactly as designed.
And they were completely useless.
The retry path this error landed on resends the same request — same units, same schema. But this failure is deterministic: the schema structure itself violates the limit, so the same input produces the same 400 every single time. The pipeline retried to its limit, burned every attempt on identical rejections, and failed anyway. That exhausted-retry failure is what surfaced as the bug.
The lesson stung a little. A retry can only fix failures where the next attempt might go differently — network hiccups, rate limits, transient provider errors. Against a structural error, retrying is just paying to watch the same rejection again. Twenty-four years of trusting retries, and I had never drawn that boundary explicitly.
The fix, in two stages
Stage 1 — route the error to the right shelf. I already had a separate retry shelf for a related class of errors: the provider's "The compiled grammar is too large." That shelf's retry is different in one crucial way — it splits the batch into smaller unit groups before retrying. Splitting changes the input: fewer units per request means fewer total unions, so the retry can actually succeed. One added string match moved the union error from the same-request retry path onto the splitting path:
retryable = errorMessage.contains("The compiled grammar is too large")
|| errorMessage.contains("too many parameters with union types");
Ugly, observational, effective. The pipeline stopped failing.
Stage 2 — count before you send. But stage 1 still burns a request to learn what I could have computed. The union limit is a published, predictable number: 16. If I count the unions in the strict schema before sending, I never enter the failure path at all. And when the schema space is user-defined, counting is the only defense: I can't test in advance against examples nobody has sent yet, but I can always count the one in front of me.
I implemented it as a capability method:
// GenerativeAICore (base class)
protected int getMaxUnionParameters() {
return -1; // no limit
}
// AnthropicV1Core (override)
@Override
protected int getMaxUnionParameters() {
return 16;
}
A recursive counter walks the schema after strict conversion — counting type arrays and anyOf branches, the exact two constructs the error message itself names — and the batching layer checks the product:
int maxUnions = core.getMaxUnionParameters();
if (maxUnions > 0 && perUnitUnions * batchLimit > maxUnions) {
batchLimit = 1; // fall back to one unit per request
}
Note what I did not do: a proportional formula like batchLimit = maxUnions / perUnitUnions. That would squeeze out more batching efficiency — and add a formula I'd have to trust in production. Binary and safe first; clever later, if ever.
And note what I could not do: pre-flight the "grammar too large" error the same way. That limit depends on unpublished internals — enum counts, string constraints, nesting depth, property totals — quantities nobody outside the provider can compute. It stays on the observational retry shelf, where splitting handles it.
That's the whole rule:
Count what you can predict. Retry what you can't.
Published numeric limits get counted before the request. Opaque complexity limits get observed, matched by error string, and absorbed by a retry that actually changes the input.
Is this an Anthropic thing? (Yes — and here's the proof)
Before shipping, I ran the same heterogeneous workload — nine unions per unit, batched — through every provider path in my system:
| Provider | Schema shape sent | Union limit | Result |
|---|---|---|---|
| Anthropic | object-keyed, nullable unions | 16 | pre-flight fires → 1-unit batches → completes |
| OpenAI (direct) | array, nullable unions | none observed | full batches, completes |
| Azure OpenAI | array, nullable unions | none observed | completes |
| Gemini / Vertex | array, subset-required — no unions manufactured
|
n/a | completes (fastest) |
| Amazon Nova (Bedrock) | object-keyed, subset-required — no unions manufactured
|
n/a | completes |
| xAI Grok | object-keyed, nullable unions — identical shape to Anthropic | none observed | completes |
The Grok row settles it. Grok receives the exact same schema shape as Anthropic — same object-keyed structure, same nullable unions — and sails through 9 × 10 unions without complaint. Same schema, different verdict. The 16-union limit is Anthropic-specific, not a property of strict decoding in general.
The Gemini and Nova rows are interesting for the opposite reason: their dialects accept required as a subset of properties, so the intermediate schema's optionality-by-omission passes straight through. No all-required rule, no manufactured unions — the constraint class is never violated because the thing it constrains is never created. Which sharpens the Anthropic row into a small irony: the same strict contract that forces the converter to manufacture nullable unions is the one that caps them at 16.
The general lesson: strict-decoding constraints are per-provider, expressed in per-provider schema dialects, and they don't transfer. The capability-method pattern keeps the special case where it belongs — one override in one provider's subclass — and every other path runs on the default -1, untouched. (Even the object-keyed vs. array split in the table is decided this way, by an older capability that checks whether a provider's strict grammar honors minItems/maxItems. getMaxUnionParameters is the second capability of its kind here, not the first.) All six paths passed with the fix in place, zero regressions.
One more thing the model taught me
While testing, I caught the model inventing a section type I never gave it. My example defined four discriminator values — table, procedure, notice, figure — and the model added a fifth, "section", for heading-only passages. Reasonable behavior — but a quiet trap. It doesn't break the schema I already sent: that schema was fixed before generation began, and a bare string type accepts "section" just fine, so no limit is tripped and nothing errors. The damage is downstream — my code branched on four types and had no case for the fifth — and one step further out. If I ever fold these real outputs back into my example to "improve" it, that new shape brings new optional keys, which become new nullable unions, which push the next request toward the 16. The model doesn't inflate the count directly; the human habit of learning from its output does.
The fix is one sentence in the prompt: enumerate the allowed values explicitly — "use only 'table', 'procedure', 'notice', or 'figure' — no other types." With that line, the same workload came back with exactly four types. If your schema derives from examples, tell the model the example is closed, not open-ended.
The model wasn't violating any published limit. It was exploiting ambiguity I had left in the prompt. That's the same lesson once more: count what you can predict, constrain what you can define, and reserve retries for the rest.
Closing
Twenty-four years of "always return a result" taught me to trust retries. This bug taught me the boundary of that trust: retries belong to the non-deterministic world. For the deterministic part — the part with a published number attached — the honest move is to count first and never fail at all.
Since writing this, I've met the harder sibling of that boundary: a failure that never announces itself at all. The request succeeds, the JSON validates, the totals reconcile — and the content is fiction. Nothing to count, nothing a retry can change. I published the measurements in July: When AI can't read, it invents — but it still sees the shape.
Built with Claude (Opus).
Top comments (0)