You ask the model for a JSON object. Your parser gets this:
Certainly! Here's the JSON you asked for:
{"severity": "high", "summary": "Login fails after password reset"}
Let me know if you'd like me to adjust anything!
json.loads() throws. Your pipeline stops. And the prompt did say "respond with JSON only."
If you build anything on top of an LLM, you've hit this. The usual reaction is to add more rules: "ONLY JSON." "NO TEXT BEFORE THE JSON." "I MEAN IT." Sometimes it helps for a while. Then it comes back.
There's a better fix, and it takes one line.
Why the model keeps doing it
Chat models are tuned to sound helpful and polite. Opening with "Certainly!" and closing with "Let me know if…" is their default register, the way they talk when nothing pushes them elsewhere.
Your format instruction is pushing, but weakly. It's one sentence the model has to remember and apply while it writes. On a short prompt that usually works. On a long prompt, or ten messages into a conversation, the default wins more and more often.
So the question isn't "how do I phrase the rule more forcefully?" It's "how do I stop needing the rule at all?"
The fix: start the answer for the model
A language model writes one token at a time, and each token continues what's already there. That's the whole trick.
Instead of describing the format, hand the model the first characters of it and ask it to continue.
❌ Describing the format:
Respond with a JSON object containing the keys "severity",
"summary" and "next_step". Do not include any text outside
the JSON.
✅ Seeding the format:
Respond with a JSON object only.
Begin your reply with exactly this text and complete it:
{"severity":
The first version asks the model to hold a rule in mind. The second gives it nothing to hold: the structure is already underway. Once a reply has started as {"severity":, a friendly sentence is no longer the natural next step. Closing the object is.
It works beyond JSON:
Want a numbered list? Seed 1.
Want a one-word verdict? Seed Verdict:
Want a table? Seed the header row: | Item | Risk | Fix |
Same principle every time: the cheapest way to get a format is to begin it.
Make it stronger: use what your API already gives you
If you're calling a model through an API rather than a chat window, check two features before you touch the prompt at all.
Structured output / JSON mode. Many providers can constrain generation to valid JSON, sometimes against a schema you supply. Where it's available, it beats any wording, because the format is enforced during generation instead of requested in text.
Response prefill. Some APIs let you write the opening of the assistant's reply yourself. Put {"severity": there and the model doesn't pretend to start with it — it literally continues from it. Not every model supports this, so check your provider's docs.
And whatever you use, validate the output in code. Models are probabilistic; treat format as something you check, not something you trust. A small wrapper does it:
python
import json
def get_json(call_model, prompt, retries=2):
"""call_model: any function that sends a prompt and returns the reply text."""
for _ in range(retries + 1):
reply = call_model(prompt)
start, end = reply.find("{"), reply.rfind("}")
if start != -1 and end > start:
try:
return json.loads(reply[start:end + 1])
except json.JSONDecodeError:
pass
prompt += (
"\n\nYour last reply was not valid JSON. "
"Reply with the JSON object only. First character: {"
)
raise ValueError("No valid JSON after retries")
Notice what the retry does: it appends the format rule at the very end of the prompt. That's not an accident, and it leads to the second half of the problem.
Two more fixes that pair with it
Seeding kills the preamble. Two other failures usually travel with it:
Hedging. "It's worth noting that this might vary…" when nothing actually varies. "Don't hedge" is too vague to work; the model decides what counts as hedging. What works better is a list of the exact phrases, grouped by where they appear: openers, hedges, closers.
Drift. The format holds for three messages, then melts into paragraphs. In long prompts, models tend to use what's at the beginning and the end more reliably than what's buried in the middle. So the rule belongs in both places: first line, and again as the last line before the model answers.
Put the three together and you have a prompt skeleton that stays clean across long tasks:
OUTPUT FORMAT: [JSON only]. ← rule at the top
BANNED OPENERS / HEDGES / CLOSERS: … ← exact phrases
[your task]
FINAL RULE: [JSON only]. ← rule at the end
Begin with: {"result": ← the seed
Try it in 60 seconds
Pick a task where your model usually pads its answer.
Run it once with your normal prompt, once with the seed added.
Compare: words before the output, hedges inside it, format at the end.
No theory to accept on faith. Just two replies side by side.
I wrote all three fixes up properly, with before/after pairs for each, the one way a ban list can backfire, and a copy-paste master template, in a free 7-page briefing:
👉 Clean Output — 3 Prompt Fixes That Stop AI Hedging (free)
If you test the seed on your own prompts, tell me in the comments what changed. I'm especially curious where it didn't work.
Top comments (0)