We replayed real cold email prompts through 7 LLMs. DeepSeek, Gemini Lite and GLM failed in a way no benchmark shows

Every email our platform sends is written by a model. Not filled in from a template: written, per recipient, from the recipient's website and a customer's brief. That makes model choice a cost line, not a taste question. A model that costs half as much per token saves real money at volume, and a model that writes 10% better emails earns real money in replies.

So in August we ran a proper comparison. Not a benchmark. A replay of our own production prompts against seven models, judged blind: GPT-5 Mini (our default), GPT-5 Nano, DeepSeek V4 Flash, Gemini Flash Lite, GLM, Kimi K2.5, and Gemini 3 Flash. GPT-5 Mini won. The interesting part is how the cheaper ones lost, because it is a failure that no public benchmark measures and that would have shipped silently to thousands of inboxes.

The setup

Our generation call is small. One prompt, one Zod schema, structured output through the AI SDK:

export async function generateAiResponseWithSchema<T extends z.ZodObject>(
  prompt: string,
  schema: T,
  capability: string,
  model: string = 'openai/gpt-5-mini'
): Promise<z.infer<T>> {
  const response = await generateText({
    model: gateway(model),
    output: Output.object({ schema }),
    prompt,
    timeout: 120_000
  })
  const parsed = schema.safeParse(response.output)
  return parsed.success ? parsed.data : (response.output as z.infer<T>)
}

The schema for an email is two strings:

export const mailToSendSchema = z.object({
  subject: z.string().min(1),
  body: z.string().min(1)
})

Because the model is a parameter and everything goes through one gateway, swapping models for a test is a one-line change. That is the whole reason this eval was cheap to run.

The recipe

Synthetic prompts tell you how a model handles synthetic prompts. We wanted to know how it handles ours, so the eval script replays production:

1. Pull real campaigns. Two paths: a local-SEO campaign whose prompt includes the recipient's actual search rankings, and generic website-based personalization, where the prompt is built by the same function production uses, from a live fetch of the recipient's website.
2. Pull real recipients. The most recently contacted recipients on those campaigns, so the inputs are exactly what the model saw in production last week.
3. Send the identical prompt to every model. Same prompt string, same schema, same call. Three runs per model, because a model that is great once and broken twice is broken.
4. Auto-check before anyone reads. Count newlines in the body. Grep for unfilled placeholders like {{firstName}}. Record latency and token cost. These checks disqualify outputs before a human wastes time judging prose.
5. Judge blind. Shuffle the surviving outputs into anonymous slots per prompt. The judge sees "A, B, C, D", not model names. Pick the best, note why.

The script writes one Markdown file with every prompt, every output, every latency, and the prompt collapsed in a <details> block so the judge can check the source when an output looks too specific to be true.

Total cost of the run: a few dollars in tokens.

What we found

The cheap models collapse newlines in structured output. This was the surprise. DeepSeek V4 Flash, Gemini Flash Lite, and GLM, asked for { subject, body } as JSON, returned bodies with zero or one newline. The email that should have been five short paragraphs came back as one block. The prose inside was often fine. The formatting made it unsendable, and nothing in the schema caught it, because a one-paragraph string is a valid string.

Our best guess at the cause: these models are trained to produce compact JSON, and the escaped \n sequences inside a JSON string are the first thing that compaction throws away. The models that held formatting were the ones with mature structured-output modes. Whatever the mechanism, the effect was consistent across all three runs, not a fluke.

This is a failure you will not see on any leaderboard. It does not show up in accuracy, reasoning, or instruction-following scores. It only shows up when you look at the actual bytes of the actual field you ship.

GPT-5 Mini won on cost and quality together. It kept formatting, filled placeholders, and wrote emails that the blind judge picked most often among the models in its price band. GPT-5 Nano, the one cheaper model that did not collapse newlines, wrote noticeably flatter prose. It is the only credible fallback if the default ever becomes unavailable, and we would accept the quality drop for an outage, not for savings.

The better writers cost twice as much. Kimi K2.5 and Gemini 3 Flash wrote emails the judge preferred more often than GPT-5 Mini. The margin was real but not large, and both cost roughly double per email. At our volume that is a meaningful line item for a modest lift in prose, with no evidence yet that the lift moves reply rate. We did not switch.

GPT-5 Mini ignores length instructions. Customers can add a custom instruction to their campaign. "Keep it short" was ignored in roughly three of four outputs. That is a prompt problem, not a model problem, and it is the one action item that came out of the eval: length has to be enforced in the schema or the prompt structure, not requested politely.

What we changed

• Kept GPT-5 Mini as the default. Every campaign that had a per-campaign model override was reset to it. We backed up the old values first.
• Made the newline check the first gate. Any future model comparison runs the shape checks before a human reads a word, and a model whose bodies come back as one block is out before judging starts.
• Two open experiments. First, replace JSON output with a plain text envelope, SUBJECT: on one line and BODY: below, and parse it ourselves. That removes the structured-output layer that seems to trigger the collapse and would make the cheap models usable. Second, make "keep it short" binding by passing a maximum paragraph count into the schema instead of the prose.

What to take from this

1. Replay your own prompts. Public benchmarks measure a distribution you do not ship. The eval that matters is your prompts, your inputs, your schema.
2. Auto-check the output shape before judging quality. Newlines, placeholders, length, language. A model that fails those has already lost. Do not let a human spend attention on it.
3. Judge blind, run three times. Model names bias judgment more than people expect. Variance between runs is a finding, not noise.
4. Price per email, not per token. Reasoning tokens count as output on most providers. A model that looks cheap per input token can be expensive per finished email.
5. A half-price model that ships broken formatting is not cheaper. It is a refund.

We run this at Deeplead, where the model writes a unique email for every recipient a customer contacts. The generation code above is production, lightly trimmed.

Naming the models means someone from those communities will push back, and the strongest pushback will be "show the numbers". The article currently says "three of four" and "roughly double" from memory. If you want that armor before publishing, say "rerun it" and I will recover the script, run the seven models on current production prompts, and put exact newline and cost figures into the post. About ten minutes and a few dollars in tokens.