The examples are the same examples. Only the wrapper changes — a JSON array becomes Markdown headings, or Markdown becomes tagged blocks — which is why teams do the rewrite in an hour and then cannot explain why the output shifted.
What the guides actually say
OpenAI’s GPT-4.1 prompting guide recommends starting with Markdown, using Markdown titles for major sections and subsections; it says XML delimiters also perform well and enable nesting; and it reports that in its long-context testing “JSON performed particularly poorly”, with a pipe-delimited form such as ID: 1 | TITLE: ... | CONTENT: ... performing well for document collections. See the GPT-4.1 prompting guide.
Anthropic’s prompting guidance takes the tag route for examples specifically: wrap examples in <example> tags, with multiple examples inside <examples> tags, “so Claude can distinguish them from instructions”, and it suggests three to five examples for best results. See Anthropic’s prompting best practices.
Two things follow. First, these are each vendor’s guidance about their own models, derived from their own evaluations, and neither generalises to the other — so “which format is best” is not a question with an answer independent of the target. Second, this guidance changes with each model generation, which is why this page is on a refresh cycle and why a house style guide that cites a vendor recommendation should cite the date as well.
Why a wrapper changes behaviour at all
The model receives one flat token sequence. Nothing in the API tells it which part is an instruction, which is an example input, which is an example output and where the examples end. Delimiters are the entire signal, and their usefulness is relative: a delimiter works when it is rare in the surrounding text and fails when it is common.
That single principle explains all of the vendor advice and most of the exceptions. XML tags segment well — unless the documents you embed are themselves XML, in which case the boundary markers are indistinguishable from content. Markdown headings segment well — unless your examples contain Markdown, which they do if your product outputs Markdown. JSON braces are common in code contexts and rare in prose, which is roughly why the same guide that reports JSON performing poorly in long context still finds it useful in coding contexts.
So the correct question when migrating is not “what does the new vendor recommend” in isolation. It is: which delimiter is rarest in my example content, among the ones the target model was trained to notice. That question has a different answer for a legal document pipeline and a code review assistant on the same model.
Three traps in the rewrite
- Escaping asymmetry silently edits your examples. JSON escapes newlines, quotes and backslashes; Markdown and tags do not. Un-escaping a JSON example turns a two-character escape sequence into a real line break, which changes what the example demonstrates. Round-trip every example through the conversion and diff against the original rendered text, not against the source.
- The output boundary becomes ambiguous. In a JSON array, where one example’s output ends is unambiguous by construction. In Markdown, a multi-line example output containing a heading or a code fence can swallow the next example. If you move to Markdown, put example outputs in fenced blocks with a fence longer than anything they contain, or keep tags for the examples and use Markdown only for the surrounding sections.
- Do not change count and format together. The temptation during a rewrite is to trim examples that no longer fit neatly. Example count changes behaviour on its own — three to five is the range Anthropic suggests, and the effect of going outside it is not the effect of the wrapper — so a combined change leaves you unable to attribute the result. One variable at a time, even though it doubles the number of evaluation runs.
A fourth consideration that is not a trap but shows up in the bill: a JSON wrapper costs more tokens per example than Markdown headings or short tags, in braces, quotes and repeated key names. With five examples in a system prompt on every request, that difference is a standing cost on every call for the life of the prompt, in the same way that tool schemas carry a standing token cost. Measure it before deciding the rewrite is not worth doing.
The model starts emitting your wrapper
The characteristic failure of a wrapper change is that the model returns the wrapper. You switch examples to tagged blocks and responses start arriving inside <example> tags; you switch to JSON examples and the model returns the whole envelope rather than the value.
The mechanism is the same segmentation logic working against you. The wrapper is the most consistent structure in the context, and the examples demonstrate that outputs appear inside it. Nothing in the prompt distinguishes “this is how examples are packaged for you to read” from “this is the shape of what you produce” unless you say so.
Two rules avoid it. Keep the wrapper distinct from the requested output shape — if the model must return JSON, do not wrap the examples in JSON. And state the output shape in its own instruction, positioned separately from the examples, rather than leaving the examples to imply it. If the output genuinely must be structured, the durable answer is to constrain it at the API level rather than by demonstration: JSON mode against structured outputs covers the difference, and which providers support what matters when the migration target is the reason you are here.
Deciding whether the change helped
Formatting changes produce small effects, which is exactly the condition under which informal comparison is worthless: run six prompts by hand and you will find evidence for whichever format you already prefer. The method has to be fixed in advance.
- Hold out a set of inputs that the prompt was not developed against — at least a hundred, drawn from production, never touched during iteration. If the set has been looked at while tuning, it is not held out and its numbers are optimistic.
- Fix the scoring function before running anything. Exact match on the extracted field where the task allows it; otherwise a rubric with the criteria written down first, applied consistently, so that two runs a month apart are comparable. Changing the rubric mid-migration invalidates every earlier number.
- Run both prompt variants over the same inputs at temperature 0, and keep the per-item results rather than only the aggregate.
- Compare pairwise: per item, does the new format win, lose or tie. Apply a sign test over the non-ties at a threshold you chose before looking.
- Apply the acceptance rule: adopt the new format only if it wins the paired test and the count of items that regressed from passing to failing is within a limit you set in advance — often zero for a formatting change, since a change of wrapper has no business breaking anything that worked. Ten wins bought with eight new failures is not an improvement, and the mean will tell you it is.
- Record the format as part of the prompt version, so the comparison is reproducible and so a later regression can be traced to it. Without that, the next person sees only that the prompt changed.
Once the choice is made, make it uniform. Mixed conventions across a prompt library are worse than either convention consistently applied, because a model that has learned your house delimiter from the system prompt has to re-segment when a section uses a different one — which is the argument for encoding the decision in a migrated style guide rather than leaving it to each author.
Both vendor recommendations quoted here are current guidance for current model families and are revised with each generation. Treat them as a starting hypothesis for your own held-out comparison, not as a setting to copy.
Top comments (0)