I ran an agent fleet for months on an empty context block

A prompt bug does not look like a bug. It looks like a model that is not very good.

I run a fleet of 240 agents against the same job. For months I read the results and drew the obvious conclusion. The work came back shallow. The fleet missed things a careful person would catch. So I did what everyone does with a disappointing model. I rewrote the instructions. I added examples. I tightened the scoring, shipped other work, came back, and rewrote the instructions again.

The instructions were never the problem. The prompt had a hole in it.

What I found

The builder assembles every prompt from a template and a payload. One slot in that template holds the context block, and that block carries everything the fleet knows about the job before it starts. The builder pulled the block out of the payload with a plain dictionary lookup and a default.

context = payload.get("context_block", "")

The harvester writes that payload. The harvester has never written a key by that name. It writes a different one. So the lookup missed, the default fired, and the builder dropped an empty string into the slot.

Python did exactly what I asked. A lookup with a default does not raise on a missing key. It hands back the default and the program carries on. Every agent in the fleet opened a prompt with a heading, a blank line, and then the next section. It parsed. It read like a prompt. It just held nothing where the facts belonged.

Nothing crashed. No log line, no traceback, no alert. A default value fails quietly, and quiet looks like success.

What it cost

I measured the fleet on the same job before and after the fix. Before, 69 of 240 agents passed. After, 223 of 240 passed. The cost stayed identical. Same model, same budget, same number of agents, same job.

The chart beside this post shows those two runs next to each other. I keep it around because the shape of it still bothers me. Nothing in the middle bought that gap. No new model, no bigger budget, no cleverer instructions. The second run just told the agents what they were working on.

Months of tuning moved that number less than one wrong key did.

Why I could not see it

I read the output and blamed the model, because the output gave me nothing else to blame. Bad reasoning and missing context produce the same symptom. The agent writes something plausible, slightly generic, a little off. You cannot tell from the text whether the model reasoned poorly or reasoned well over nothing.

That is the trap. My eyes were on the results the whole time and the results could not tell me. The answer sat in the prompt I sent, and I never once printed it.

What I changed

I stopped giving required slots a default. The builder raises now when a key it needs is missing, and it refuses to send any prompt whose required slot came back empty. A loud failure on the first run beats a quiet one across months of runs.

Then I logged the assembled prompt for one agent per batch and read it myself. That took about a minute. It would have caught this on day one.

Treat a default as a claim. You are saying absence is fine here. For anything the work depends on, absence is not fine, and the language will never tell you so. Write the check yourself, then go read what you actually sent.