The model obeys your schema, not your description

Two models. Same prompt, same tool description, same request. One of them returned this:

{ "kind": "entity", "entityName": "todo", "definition": { "fields": { "title": "text" } } }

The other returned this:

{ "kind": "entity", "name": "todo", "fields": { "title": "text" } }

The second one is wrong, and our downstream patcher rejected it with a 422 that told nobody anything useful. What took me a while to work out was why the first model got it right, because the answer turned out to have nothing to do with being smarter.

This was May 2026, Opus 4.7 and Sonnet 4.6 at the time. The story generalizes to whatever pair of models you're holding today.

The setup

We have a tool called apply_patches. An LLM reads a user request plus a source file and emits a list of structural change operations: add this entity, replace that handler, remove that metric. Each operation carries a pattern, the canonical object form of the thing being changed.

The tool schema for that pattern parameter was, in effect:

{ "type": "object", "properties": { "kind": { "type": "string" } }, "required": ["kind"] }

kind is a string and everything else is whatever. The actual shape lived in the tool description, a paragraph of prose with examples, the way most people write tool definitions.

The big model complied anyway. The smaller one didn't. Both had read the same description.

Why the big model complied

It had seen the shape before.

entityName and definition.fields are our field names, from our framework. To a model with training exposure to that shape, "emit an entity pattern" retrieves a memory. To a model without it, "emit an entity pattern" is a guess from the description text, and if you're guessing what an entity looks like, { name, fields } is a better guess than the truth. It's what everyone else's API would call those things.

So this is an exposure gap rather than a capability gap, which matters because you can't fix an exposure gap by paying for a bigger model. It will show up for any model on any shape that isn't in its training data, which is to say on your proprietary shapes, indefinitely. The less your schema looks like the rest of the internet, the harder your description has to work. And descriptions are not what the model is validated against.

The tool schema is. So we moved the contract into it.

Tight on the common kinds, loose on the tail

We have around twenty pattern kinds. Nine of them account for roughly 85% of everything the model emits. The other dozen (relation, workspace, secret, claimKey, systemScope and friends) show up rarely.

Writing strict schemas for all twenty would have been a week of work and a permanent maintenance tax, so we didn't. Each common kind became a discriminated oneOf branch with a real required list:

{
  title: "EntityPattern",
  properties: {
    kind: { const: "entity" },
    entityName: { type: "string" },
    definition: { type: "object" },
  },
  required: ["kind", "entityName", "definition"],
}

The long tail got one fallback branch that requires nothing but kind:

{
  title: "OtherPattern",
  properties: {
    kind: {
      type: "string",
      not: {
        enum: [
          "entity", "requires", "toggleable", "nav", "writeHandler",
          "queryHandler", "hook", "notification", "metric",
        ],
      },
    },
  },
  required: ["kind"],
}

Rare kinds still go through unvalidated at the schema layer, and the runtime patcher catches them. That split, tight on the discriminator values you see constantly and permissive on the ones you don't, is the part worth stealing. It costs an afternoon instead of a week and it targets the failures you actually get.

We did the same for the natural keys that replace and remove operations use, and pinned the per-operation requirements with allOf plus if/then, so the model can't hand us a replace with nothing to replace:

allOf: [
  { if: { properties: { op: { const: "replace" } } }, then: { required: ["id", "pattern"] } },
  { if: { properties: { op: { const: "add" } } },     then: { required: ["pattern"] } },
  { if: { properties: { op: { const: "remove" } } },  then: { required: ["id"] } },
]

Both Anthropic and OpenAI honor oneOf and allOf/if/then in tool input schemas. Most people skip them because the flat version works well enough on whichever model they tested with.

Did it work?

The evidence is thinner than I'd like, and it points the right way.

We ran three fixtures live, twice, for about $0.18 total. Before the change: two passed, one failed. The failure was the rename-entity case, emitting { kind: "entity", name, fields }. After: three passed, and the fixture that had been failing emitted { kind: "entity", entityName: "todo", definition: { fields: ... } }, byte for byte the shape the big model had been producing all along.

Three fixtures is not a benchmark. What convinced me was which failure disappeared and what replaced it. The smaller model stopped inventing field names and started producing the canonical shape, on the exact case that had been failing.

The real payoff came later. Once the smaller model could reliably emit the structured shape, it became viable as the default. That's usually the whole business case for schema work: a tight schema is what makes the cheap model good enough.

Footgun 1: oneOf is strict XOR

Two weeks later, a code review caught something the tests hadn't.

For replace and remove operations we have a parallel set of variants describing just the natural key. One of them is a singleton fallback: a kind and nothing else. And { kind: "entity" } matched both the entity branch and the fallback branch.

oneOf means exactly one. Two matches is a violation. Anthropic tolerated it; OpenAI's strict mode rejects the schema outright. Same schema, one provider silently fine, the other refusing to run.

The fix is the not.enum you saw above, where the fallback explicitly excludes every kind that has its own branch. Worth internalizing if you're building discriminated unions in JSON Schema: a fallback branch is not automatically disjoint from the specific ones. You have to make it disjoint by hand, and a test has to hold it that way, because the day someone adds a tenth specific branch and forgets the exclusion list is the day one of your providers starts 400ing.

Footgun 2: field order is a token budget problem

Different bug, same week. A generate_feature call came back with featureName, packageDescription, rationale, and no source. Source being the entire point of the call.

The model had emitted rationale first, written about 700 tokens of thoughtful design commentary, and hit maxTokens: 4000 before it got to the field that mattered. Nothing errored. We just got a well-argued explanation of a file that didn't exist.

Three fixes, in descending order of how much I trust them:

• source.minLength: 100 and rationale.maxLength: 600, plus a blunt "BRIEF, 2-3 sentences" in the description. These are real constraints and they're what actually holds.
• Raised maxTokens to 8000 for this one tool. The bounded tools stayed at 4000, since an unbounded budget everywhere just makes the truncation rarer and weirder.
• Declared source first in the schema. Anthropic has been observed to emit arguments in declaration order. Observed, not documented. It costs nothing, it might help, and if a provider update changes it tomorrow nothing fails loudly. I left a comment in the source saying exactly that, and you should treat it the same way: a hint, not a contract.

Scores on the affected fixture went from 0.50 to 0.85, on both models, which is the tell that this was never a model-quality issue. The remaining 0.15 is genuine content quality: the model doesn't reach for one of our helpers when it should. That's a prompt and few-shot problem, and no amount of schema work will fix it. Knowing which of your failures are schema-shaped saves you from tightening things that were never loose.

The part nobody solves for you

These schemas are a hand-written mirror of our framework's real pattern types. Two definitions of the same shape, in two files, with nothing but discipline between them. When someone adds a required field to the real type, the schema doesn't know.

We pin what we can in a contract test: the number of variants, the required-field list per kind, the not.enum exclusion list. Drift fails a test instead of confusing a model in production six weeks later. That's a smoke alarm rather than a solution. If you generate your tool schemas from your actual types, you're ahead of us. If you're hand-writing them like we are, at least pin them.

What it costs

The obvious objection: you just made your prompt bigger, on every single call.

About 3KB bigger, in our case. With prompt caching that's one cache write at 1.25× input rate, roughly $0.0002 for the schema chunk, and every subsequent call in the cache window reads it at 10%. The schema sits in the stable prefix, which is exactly where caching is designed to put it.

Structured output used to carry a real tradeoff between schema size and bill. With caching it mostly doesn't. If token cost is what's keeping your tool definitions vague, go measure it. The number is probably smaller than the cost of one confusing 422 in production.

The short version

• The description is advice. The schema is the contract. Models validate against one of those.
• A model complying with your undocumented shape may just be recognizing it from training. That's not a capability you can rely on, and your next model isn't guaranteed to have it.
• Tight schemas on the handful of kinds that dominate your traffic, one loose fallback for the tail. Don't schema the whole universe.
• Make your fallback branch explicitly disjoint, or oneOf will bite you on the strictest provider you support.
• Length constraints hold. Field order is a hint. Know which is which.
• The reason to do any of this is usually that it makes the cheaper model good enough.

Comments

[Tae Kim]

The 85/15 tight-on-frequent approach matches what I landed on for structured extraction in a RAG pipeline. Going tight on all the types took two days and kept breaking when underlying types changed, while putting discriminated branches on the five or six dominant shapes got to comparable compliance faster. The field-order-as-hint note is the one I'd add to the codebase too; I've seen that ordering behavior shift between patch versions on the same provider, which is the whole reason the comment you left is the right call.