Ryan Cole

Posted on Oct 12

Stop writing prompts like prose: the 4-part contract that kills LLM drift

#ai #llm #programming #productivity

Most "prompt engineering" advice is about phrasing. The real problem is that people write prompts the way they write emails — as prose — and then act surprised when the model "drifts" between runs.

Production prompts aren't prose. They're contracts. A contract has to define four things explicitly, or the model fills the gaps with whatever is statistically likely — which is exactly the drift you're seeing.

The four parts

Every prompt that has to behave the same way twice should answer these four questions:

Role — who is speaking, and what do they optimize for?
Task — what's the single deliverable? (Not a list. One thing.)
Constraints — what must never happen?
Output contract — the exact shape of a valid answer, and what to do when the input can't satisfy it.

If a prompt is missing any of these, the model guesses. Guessing is the enemy.

Before: prose

You are a helpful assistant. Summarize the following customer review and tell me
if it's positive or negative and maybe what the main complaint is. Be concise.

This fails in five different ways depending on the input, and you can't predict which.

After: contract

ROLE
You extract structured feedback from customer reviews. You optimize for accuracy,
not for making the review sound better.

TASK
Given one review, produce a single JSON object classifying sentiment and listing
complaints.

CONSTRAINTS
- Never invent a complaint that is not present in the text.
- Never output anything except the JSON object. No preamble, no markdown fence.
- If the review is not about a product, set "relevant": false.

OUTPUT CONTRACT (strict)
{
  "relevant": boolean,
  "sentiment": "positive" | "neutral" | "negative",
  "complaints": string[],   // verbatim or closely paraphrased, [] if none
  "confidence": number      // 0..1
}
If you cannot comply, output {"error": "<short reason>"}.

The second prompt is longer, and that's the point. Every extra line is a decision you're making once instead of letting the model make differently each run.

The output contract is the part everyone skips

A schema without an escape hatch causes a specific failure: when the input can't fit, the model fabricates to satisfy the schema. It would rather return a wrong-shaped-but-valid object than admit defeat. So always define the failure shape explicitly ({"error": ...}), and validate it:

import json

REQUIRED = {"relevant", "sentiment", "complaints", "confidence"}

def validate(raw: str) -> dict:
    obj = json.loads(raw)              # raises -> caller retries/fails over
    if "error" in obj:
        raise ValueError(obj["error"])
    missing = REQUIRED - obj.keys()
    if missing:
        raise ValueError(f"missing keys: {missing}")
    if obj["sentiment"] not in {"positive", "neutral", "negative"}:
        raise ValueError("bad sentiment")
    return obj

Now "drift" becomes a loud, catchable error instead of a silent wrong answer three systems downstream.

Constraints that actually change behaviour

The constraints line is where you win. Weak constraints do nothing ("be accurate"); strong constraints remove specific failure modes:

"Never invent a complaint not present in the text" → stops hallucinated details.
"No preamble, no markdown fence" → stops the model wrapping JSON in json ... .
"One deliverable" → stops the model answering three questions you didn't ask.
"If X, do Y" → gives the model a legal exit, which reduces fabrication.

Notice they're all negative or conditional. "Be helpful" is unfalsifiable; "never invent" is testable.

Gotchas

Test the contract, not the vibe. Keep 20–50 input/expected pairs and run them every time you touch the prompt. A prompt change that "feels better" can silently break 10% of cases.
Don't over-constrain the reasoning. Give the model room to think if the task needs it; just don't let that thinking leak into the output. (Use a two-call pattern: one to reason, one to format.)
Version your prompts like code. A prompt is a dependency. If it's not in git with a changelog, you can't roll back the regression.
Same contract, every provider. If you fail over between models, the output contract is what keeps their responses interchangeable.

Why this matters

The gap between a demo prompt and a production prompt isn't cleverness — it's explicitness. A contract you can validate is a prompt you can actually ship, monitor, and debug. Everything else is hoping.

I collected 32 of these production-grade system prompts (six categories, each with a strict output contract) plus JSON schemas and a stdlib-only validator CLI here:

https://ancuboy.gumroad.com/l/promptops-pro

And a bigger bank of prompt templates for creators/e-commerce is here (pay what you want, $1 min):

https://ancuboy.gumroad.com/l/ai-ultimate-prompt-vault

Feedback welcome — especially if you have a constraint that reliably kills a failure mode I didn't list.

DEV Community