- Book: Prompt Engineering Pocket Guide: Techniques for Getting the Most from LLMs
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
"Add a few-shot example" is the most-given prompt advice on the internet. Stack Overflow. Twitter. Half the prompt-engineering YouTube channels. It worked great in 2022.
It's now actively wrong for a chunk of real production tasks. The example doesn't just stop helping. It drags accuracy down, blows up your token bill, and biases the output toward whatever shape the example took. You ship the regression and blame the model.
This post is about catching that before the regression hits prod.
The 2022 advice that became cargo cult
Back when GPT-3 was the frontier, few-shot was a small miracle. The base model didn't understand task framings out of the box. You had to show it what the answer should look like. The 2020 paper that coined "language models are few-shot learners" wasn't being cute. That was the actual capability gap.
Three model generations later, things changed. Instruction-tuned models (GPT-4o, Claude Sonnet 4, Gemini 2.5) understand "extract every date from this text and return JSON" without an example. Reasoning models (o-series, Claude Opus with extended thinking, Gemini's thinking variants) go further. They often do better when you describe the task and step back. The example becomes noise the model has to disambiguate from the actual input.
The advice didn't update. People paste the same "here's an input, here's an output" pattern into Claude 4.7 prompts because a 2022 blog post told them to. Then they wonder why their evals look weird.
Three task shapes where examples hurt
Not every task. Few-shot still earns its place in narrow classification, novel output formats, and a couple of math patterns (more on those later). But three shapes lose accuracy when you add examples, and they're three shapes that show up constantly in production.
High-recall extraction. Pull all entities of a class from a document. Names, dates, line items, claim references, transaction IDs. The failure mode: your one example pulls 4 dates from a 200-word passage. The model implicitly learns "this task returns about 4 things" and stops at 4 even when the actual document has 11. Recall drops. The fix isn't a longer example. That just moves the anchor. It's no example at all.
Creative generation. Marketing copy, alt text, summaries with a specific voice. The example is now the most concrete thing in the prompt, so the model copies its rhythm, sentence length, and vocabulary. You wanted variation; you got 50 outputs that all sound like the example. Worse, A/B tests on your landing page start showing the model's example phrasing leaking into production copy verbatim. The fix: describe the voice in prose, then trust the model.
Strict-format instruction-following. "Output JSON with fields A, B, C; no preamble; no markdown fences." The schema-and-no-example version of this works well in Claude 4 and GPT-4o. Add a one-shot example and the model starts copying structural choices from the example: quote style, field order, whether it wrapped the JSON in a code fence. If your example had a trailing newline, half your outputs now have a trailing newline. You hard-coded a defect.
The pattern across all three: the example over-anchors. The model treats it as a precedent instead of a hint. On a model that already understands the task, that's a regression.
Why frontier reasoning models flip the few-shot calculus
Reasoning models think before they answer. They allocate hidden tokens to plan, sketch, revise. When you hand them a few-shot example, they spend reasoning budget imitating the example structure instead of solving the actual problem. You can watch this in trace tools. The thinking trace will reference the example, try to mirror it, then constrain its own output to match.
For tasks where you want the model to reason from scratch, that's a tax. The o-series docs explicitly call this out: minimise few-shot prompting, prefer clear task descriptions. Anthropic's prompt-engineering guide for Claude 4.x says the same about extended thinking. Examples can short-circuit the reasoning chain.
This isn't a small effect. On a math-with-formats benchmark I'd seen a team run internally, switching from a 3-shot prompt to zero-shot on Claude Opus extended-thinking moved accuracy by 6 points. The few-shot version was the version they'd been shipping.
The 30-line ablation that proves it on your own task
Don't trust this post. Don't trust the model card either. Run the ablation against your actual task, your actual data. Thirty lines of Python:
import asyncio
import json
from anthropic import AsyncAnthropic
client = AsyncAnthropic()
MODEL = "claude-sonnet-4-5"
ZERO_SHOT = "Extract every date mentioned in the text. Return JSON array of ISO-8601 strings. No preamble."
FEW_SHOT = ZERO_SHOT + """
Example input: "We met on March 3rd 2024 and again on the 7th."
Example output: ["2024-03-03", "2024-03-07"]"""
async def run(prompt: str, text: str) -> list[str]:
msg = await client.messages.create(
model=MODEL,
max_tokens=512,
messages=[{"role": "user", "content": f"{prompt}\n\nText:\n{text}"}],
)
return json.loads(msg.content[0].text)
async def main(eval_set: list[dict]) -> None:
for variant, prompt in [("zero", ZERO_SHOT), ("few", FEW_SHOT)]:
preds = await asyncio.gather(*[run(prompt, row["text"]) for row in eval_set])
hits = sum(set(p) == set(r["truth"]) for p, r in zip(preds, eval_set))
print(f"{variant}: {hits}/{len(eval_set)} exact-match")
asyncio.run(main(json.load(open("eval.json"))))
That's the whole thing. Point it at 50 labelled examples from your real traffic and you'll see the gap (or the lack of one) in under a minute of wall time. The shape of the answer matters more than the absolute score. If zero-shot wins by 4 points across 50 samples, your few-shot prompt is costing you accuracy and tokens.
Two notes on running it honestly. First, vary the example. A bad few-shot result might be a bad example, not a bad pattern. Try three different examples; if all three lose to zero-shot, the pattern's wrong for the task. Second, look at the wrong-answers, not just the score. If few-shot fails by under-recalling and zero-shot fails by hallucinating, those are different problems with different fixes.
When examples DO help
Few-shot still earns its keep on three task shapes. Don't throw it out wholesale.
Narrow-domain classification with non-obvious labels. You're labelling support tickets into 14 internal categories with names like T1-billing-disputed-charge. The model has no idea what your taxonomy means from the label alone. Two examples per class teaches it the boundary; even one-shot suffices for the obvious labels. Accuracy jumps from "guessing" to "useful."
Novel output formats. You need output in a custom DSL nobody's seen before. Configuration for an internal rules engine. A bespoke query language. Show, don't tell. Two examples beat a 400-word spec because the model parses examples faster than prose.
Few-shot chain-of-thought for math. The "let's think step by step" trick combined with a worked example. Still works. The example anchors the reasoning style, not the answer shape, and on math that's a win. Reasoning models have largely absorbed this, but for the non-reasoning tier (Sonnet without thinking, GPT-4o-mini) it still moves numbers.
If your task isn't one of those three, default to zero-shot and add examples only when an ablation proves they help.
Example length matters more than count
Here's the part most teams miss. The damage scales with example length, not example count.
A five-shot prompt where each example is 15 tokens is fine. A one-shot prompt where the single example is 800 tokens (a full document with its full extraction) is the prompt that quietly ruins your accuracy. The model has to hold 800 tokens of "this is what good looks like" in attention while processing the actual input. The signal-to-noise ratio collapses.
The token economics push the same direction. Examples are not cached unless you pin them with prompt caching (Anthropic's cache_control, OpenAI's automatic prefix caching). If your one-shot example is 800 tokens and you send a million requests, you've spent eight hundred million tokens repeating the same example. That's a five-figure invoice line that may be hurting accuracy too.
Two practical rules:
- Cap each example at roughly 10% of the model's effective context window for the task. If you're processing 2k-token inputs, examples stay under 200 tokens each. Trim aggressively. The model doesn't need the full document; it needs the input-output shape.
- If you can't compress the example below that cap, the task probably wants a different technique. Schema-guided decoding, structured output APIs, or a fine-tune all beat "huge sticky example" on production economics.
Prompt caching is the workaround when you genuinely need long examples. Set cache_control: {"type": "ephemeral"} on the example block, and after the first request the example is amortised across requests until the cache TTL expires (5 minutes default on Anthropic). It cuts the cost but doesn't fix the attention dilution. Cached or not, an 800-token example still drowns a 200-token input.
The takeaway
Few-shot is a tool, not a default. The 2022 advice was right for 2022 models. Three model generations in, it's a tool to reach for after you've measured, not before.
Three rules to ship with:
- Default to zero-shot. Add examples only when an ablation proves they help on your task.
- For reasoning models, lean even further toward zero-shot. The example competes with the thinking trace.
- Example length matters more than count. A long example is a tax on every request you send.
The thirty lines above will pay for themselves the first time you delete a 600-token few-shot block that was costing you points and dollars at the same time.
What's a prompt where deleting the examples made the output better on your task? Drop the before/after numbers in the comments.
If this was useful
The Prompt Engineering Pocket Guide digs into when each prompting technique pays its rent: zero-shot, few-shot, chain-of-thought, self-consistency, and the structured-output patterns that quietly replaced half of them. The chapter on ablation discipline is the one that maps directly to this post: how to run the test, how to read the wrong answers, and how to know when "best practice" became cargo cult on your specific task.
Top comments (0)