Your prompt works in the playground and dies in production

The prompt was never the problem

I spent a whole afternoon last month tuning a prompt. It was beautiful. In the playground it fixed a broken function on the first try, explained the bug, and even suggested a test. I shipped it into our pipeline feeling smug.

The next morning it was wrong four times out of five.

Same model. Same temperature. Same carefully worded prompt with the bold "you are an expert Python engineer" opening line. And yet the playground version was a genius and the production version was guessing. I did what everyone does. I rewrote the prompt. Added more instructions. Told it to think step by step. Threatened it politely. Nothing moved the needle.

Then I actually looked at what my code was sending the model, and the answer was embarrassing. In the playground I had pasted the whole file. In production I was sending three lines.

That is the gap. Almost every time. The words you argue over matter far less than the stuff you forgot to include.

Before and after, with a real task

Here is roughly the request that was failing. Imagine a helper that parses a date range from a query string.

Fix this function. It throws on some inputs.

def parse_range(q):
    start, end = q.split("..")
    return date.fromisoformat(start), date.fromisoformat(end)

The model will happily "fix" this. It might wrap it in a try/except. It might add a check that .. is present. It might validate that start is before end. All reasonable. All possibly wrong, because it has no idea which failure I actually care about. It is inventing a spec.

I get a different answer every run because there is nothing to anchor it. The prompt is fine. The context is empty.

Now the same task with the missing pieces bolted on.

Fix parse_range so this failing test passes.

def parse_range(q):
    start, end = q.split("..")
    return date.fromisoformat(start), date.fromisoformat(end)

Failing test:

def test_open_ended():
    # "2026-01-01.." means from that date to today
    s, e = parse_range("2026-01-01..")
    assert s == date(2026, 1, 1)
    assert e == date.today()

Traceback:
ValueError: Invalid isoformat string: ''

Constraint: date is imported from datetime. Do not add new imports.

Now there is exactly one correct answer, and the model lands on it almost every time. The open-ended range case was never in the prompt before. No amount of "be careful" would have surfaced it. The test carried the requirement that my English left out.

Notice what changed. I did not make the instruction smarter. I gave it the failing test, the real error, and one boundary condition. The clever version guessed. The boring version knew.

Why the playground lies to you

The playground feels honest because you are pasting context by hand without noticing. You grab the whole file. You include the error you just saw in your terminal. You mention the one edge case that is on your mind. All of that is context, and none of it survives the trip into your application code, where you are usually stuffing in a truncated snippet and a template string.

So the playground is not testing your prompt. It is testing your prompt plus everything in your short term memory. Production only gets the prompt.

The failure modes are boring once you know to look for them. The model does not see the imports, so it invents a library. It does not see the caller, so it changes a return type that breaks three other functions. It does not see your error message, so it fixes a bug you do not have. It does not know today's date, so it hardcodes one. Every one of these looks like the model being dumb. It is you sending a postcard and expecting a novel back.

A checklist for supplying context

Before you touch the wording again, walk this list. It has saved me more hours than any prompt trick.

• Send the surrounding code, not the snippet. The function, its callers, its imports, the types it touches. If the model has to guess the shape of an object, it will guess wrong.
• Include the actual failure. The real traceback, the failing assertion, the log line. "It throws sometimes" is not a bug report and the model cannot act on it either.
• State the constraints out loud. Which library, which version, what not to import, what must stay backward compatible. Unwritten rules get broken.
• Pin anything time or environment dependent. Today's date, the runtime version, the OS, feature flags. Models do not know when "now" is.
• Show one example of right. A passing test, a sample input and its expected output. One concrete example beats three paragraphs of description.
• Cut the noise. Context is not "paste the entire repo." Irrelevant files bury the signal and cost you tokens. Send what touches the task and stop.
• Diff the two payloads. When it works in the playground and fails in prod, print exactly what your code sent and compare it to what you pasted by hand. The difference is your bug, every single time.

The pattern under all of this is simple. When the output is wrong, ask what the model could not see. Fix that first. Rewrite the prompt only after the context is complete, which, honestly, is usually never.

Good prompting is mostly good context assembly. The words are the last five percent.

AGINE Academy is an independent product by AGINE AI (not affiliated with Anthropic). We teach building with Claude by doing the work, not watching lectures.