Structured output gotchas across 32 LLM providers (and how to test for them)

Structured output gotchas across 32 LLM providers (and how to test for them)

We route 32 models behind one OpenAI-compatible key at heypico.ai, which means we see every provider's idea of "JSON mode" up close. They are not the same.

The gotchas that actually bit us

1. Trailing commas and prose wrappers. Some models emit clean JSON in JSON mode; others still wrap objects in markdown fences even when told not to. We now strip fences before parsing.

2. Enum drift. You ask for status: "ok" | "error"; some providers return SUCCESS. A schema check with explicit enum coercion fixed a production bug.

3. Function-calling shape differences. Arguments arrive as a JSON string on some providers, an object on others. Normalize at the boundary.

4. Refusals disguised as output. A model that refuses a task sometimes still emits valid-but-wrong JSON. Test refusal paths, not just happy paths.

The harness

Every prompt ships with a schema + a 15-case test set. CI runs it against 3 models minimum (one frontier, one cheap, one open-weights) before merge. It is boring and it catches real bugs.

If you want the multi-model part without the plumbing: HeyPico gives you 32 models behind one OpenAI-compatible key. Singapore-based, CASA Tier 2, free trial for the first 100 devs.