Ploy builds production marketing websites with an AI agent. They've been benchmarking every frontier release for months. Nothing beat Claude Opus until GPT-5.6 Sol — 2.2× faster, 27% cheaper, better visual scores.
Then they actually tried to ship it. That's where it got interesting.
"We use Vercel's AI SDK, but switching from Claude Opus 4.8 to GPT-5.6 Sol still exposed provider-specific assumptions throughout our stack."
What the numbers look like
| Claude Opus 4.8 | GPT-5.6 Sol | |
|---|---|---|
| Cost (per build) | $3.06 | $2.22 |
| Wall-clock time | 8m 00s | 3m 42s |
| Input tokens | 2.60M | 1.70M |
| Visual score | 0.936 | 0.970 |
GPT-5.6 also wrote significantly less code to get there — 2,508 characters of CSS against Opus's 17,957. Fewer tokens, less output, better result.
Fix your eval harness before you trust a single number
Before any of that was usable, Ploy had to fix their evaluation suite.
Tool-call budgets were sized for Opus's sequential style. GPT-5.6 makes parallel calls and exceeded those budgets on cases it was solving correctly. The eval executor didn't support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. One dataset was missing an explicit minScore threshold, silently defaulting to 1.0 — so GPT-5.6 "failed" a hero image that scored 0.98.
About a third of raw failures in the first run came from harness assumptions, not model behaviour. Those failures were unevenly distributed in ways that would have made GPT-5.6 look worse than it was.
The lesson: if you're evaluating a challenger against an incumbent, triage the traces before trusting the pass rate. An eval that rewards the new model for behaving like the old one isn't measuring capability — it's measuring familiarity.
Tool schemas need rethinking per-provider
Ploy's code tool has 25 parameters. Claude sends the two or three it actually uses. GPT-5.6 sends all 25, filling unused slots with plausible-looking invented values (offset: 0, timeout: 120000, siteId: "00000000-...").
That sounds like a cosmetic annoyance. It wasn't. The tool implementation couldn't distinguish invented values from real ones, so 52–64% of file reads returned empty — but the tool still returned success: true. The model saw a success, read a blank result, compensated with more calls, and degraded.
Prompting didn't fix it. strict mode didn't fix it either (and would've required stripping schema validation). The working fix: a schema transform at the provider boundary that rewrites optional parameters as anyOf: [T, null]. GPT-5.6 sends explicit nulls for unused params, which get stripped before the tool runs. Empty reads dropped from 52% to 0%.
Prompt caching is a different API, not a different knob
Before fixing caching, GPT-5.6 appeared ~50% more expensive than Opus. After fixing it, it came out cheaper. The entire cost gap was cache misconfiguration.
The key difference: Anthropic caches at the organisation level — any conversation can hit a shared prefix entry. GPT-5.6 dropped implicit partial-prefix matching. Without explicit prompt_cache_key setup, a new conversation sharing Ploy's 29K static prefix cached 0% of it and paid the full uncached rate every time. GPT-5.6 also applies a 1.25× surcharge to every uncached prompt.
Ploy ended up with per-workspace cache keys and a layered breakpoint structure — static tools/prompt as one entry, workspace context as another, per-session context as a third. First-call hit rates went from ~0% to 83.7%. Total uncached input tokens fell 28%.
The third fix: reasoning replay
One more gotcha. GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default. Production conversations were intermittently failing with Item 'rs_...' not found. Setting store: false makes the SDK request encrypted reasoning content and replay self-contained blobs instead of server pointers. Worth checking if you're seeing intermittent context errors.
What to do
- Evaluating GPT-5.6 against your current model? Audit your eval harness first — tool budgets, implicit thresholds, anything Opus-specific. Trust traces over aggregate pass rates.
- Complex tool schemas? Check whether your implementation can handle fully-filled params. The nullable schema transform is the cleaner fix over prompting.
- Running prompt caching? Test your actual cache hit rate at the API level. The providers cache differently — a config that works on Anthropic may hit 0% on OpenAI.
-
Using the Responses API with reasoning? Set
store: falseto avoid server-state pointer failures in long conversations.
The model benchmarks are real. So is the migration work. Plan for both.
Source: Ploy — Migrating a production AI agent to GPT-5.6
✏️ Drafted with KewBot (AI), edited and approved by Drew.
Top comments (0)