Here is how I used to change a prompt, and I suspect it's how you do it too.
Take a handful of real inputs — six felt like plenty, you can hold six in your head. Run the old prompt and the new prompt over them. Put the outputs side by side and read. One of them is visibly better. Ship that one.
This feels like evaluation. It has inputs, a comparison, and a decision. It is not evaluation, and it took me three consecutive prompt rewrites to notice.
The check that ended it
On a whim, I ran the same prompt twice over the same six inputs. Same model, temperature turned well down, nothing changed between the runs but the clock.
The two runs of one prompt disagreed with each other about as much as the two competing prompts had. Average output length moved by a quarter. And the single example I'd been treating as proof that version B was better — one clean, sharp result the other version had fumbled — was produced by the other version the second time around.
So every one of those confident side-by-side reads had been a coin flip that I narrated as judgment. Three rewrites in a row, each one carefully tuned to the previous run's dice.
If you don't know your run-to-run spread, you cannot tell a difference from a draw. A low temperature is not zero. Batching, sampling, and the order things happen to arrive in all wobble the output, and the wobble at small sample sizes is bigger than most prompt edits. Measure your own noise floor first, by running one prompt twice and comparing it to itself. That number is the bar every future comparison has to clear.
Vibes don't aggregate
Once the sample has to be tens of items rather than six, reading them stops working. You can hold six comparisons in your head. You cannot hold thirty, and by item twenty you are no longer applying the standard you started with — you're tired, and tired reviewers get generous.
So the checks have to be things a regex can count. Mine, for a summarizer:
- Contains a digit. Crude proxy for "carries a concrete detail" rather than a vague gesture at the topic.
- Repeats the headline. Share of headline words reappearing in the summary; over ~70% and the summary has added nothing.
- Filler opener. Starts with "The article describes…", "The author discusses…", or contains a stock phrase like "key aspects".
- Mixed scripts inside one word. A specific failure of small models translating between alphabets, where half a word gets translated and half doesn't.
- Untranslated title. Not a single character of the target script — the model silently skipped the job.
Every one of these is obviously reductive. "Contains a digit" is not "is a good summary." That's fine, and it's the point:
A metric doesn't have to capture quality. It has to capture a failure you can name. I can't write a regex for "reads well." I can write one for each of the five specific ways the output had actually embarrassed me in production, and the sum of those five moving in one direction is a real signal about a change. Quality is what's left after your known failures stop happening.
The corollary is that you have to have a list of named failures, which means the honest first step isn't writing a harness at all — it's going back through the bad outputs you've already seen and naming what was wrong with each one.
Aggregates lie; count in pairs
Percentages across two runs invite the same coin-flip reasoning at a larger scale. The fix that made me trust the result was cheap: fix the sample so both versions see byte-identical inputs (seed the query, don't re-sample), then count per item — this flaw appears only in version A's output for this input.
Aggregate says version B paraphrases the headline more often. Paired count says: on these specific inputs, B paraphrased where A didn't, several times over, and the reverse happened once. When the aggregate and the paired count point the same way, it's a signal. When they disagree, you're still inside the noise and the honest answer is "no difference found."
The result I didn't expect
The change I was measuring was a tightening. The old instruction was wordy: add exactly one detail the headline doesn't already have — a number, a name, a mechanism, a limitation — plus a list of banned phrasings and a written-out example of a bad output. I'd replaced all of it with something short and forceful: start immediately with the fact.
Shorter, firmer, unambiguous. It lost, and not narrowly. The terse version produced fewer concrete details and close to twice as many summaries that merely restated the headline in different words.
Reading the failures, the mechanism is obvious in hindsight. "Start immediately with the fact" doesn't tell the model which fact, and the most available fact in the context is the headline it was just handed. So it restated the headline — immediately, and as a bare fact, exactly as instructed. The wordy version won because it named the class of thing to add and showed a worked example of the failure mode.
A stricter instruction is not a stronger one. Emphasis — caps, "must", "always" — costs nothing to write and does very little. A concrete example of the output you don't want does much more work than any amount of insistence about the output you do.
And the fix wasn't in the prompt at all
The real reason those summaries kept paraphrasing headlines turned out to be upstream. The model was being shown the opening slice of each article — first N characters, the obvious way to fit a long text into a context budget.
Abstracts and press releases put the framing at the front and the result at the end. Truncating from the front hands the model everything except the finding. It wasn't being lazy or dumb; it genuinely did not have the answer in front of it, and restating the headline was the best available move.
Keeping the head and the tail and dropping the middle — where the methodology lives — beat every prompt rewrite I'd attempted, including the ones I'd "won" by reading six examples.
Before you rewrite the prompt again, check whether the answer was in the context at all. Prompt engineering is the fun part and context assembly is the boring part, which is roughly why the bug lives in the boring part.
The parts that are just bookkeeping
Two things I'd tell my earlier self.
Write the harness as a throwaway script and keep it. Mine reconstructs both prompt variants from the current source and asserts that the reconstruction contains the strings it should, so a comparison can't quietly measure two copies of the same thing. That assert has already saved one run.
And watch what your speed metric is actually made of. I had been tracking characters per second, which fell after a change that made things better: the prompt got longer, the outputs got shorter and denser, so the ratio dropped while the time to process one item improved. Measure per unit of work you care about, not per unit of output the model happens to emit — otherwise you'll optimize for verbosity and call it throughput.
None of this is clever. It's the boring statistical hygiene that every other part of engineering already takes for granted, applied to a component whose output happens to be prose. The reason we skip it for prompts is that prose is readable, and readable feels like measured. It isn't. I lost three iterations to that, on a summarizer that goes out to real readers in a news digest I run, and the only thing I regret is not running the same-prompt-twice check on day one. It costs one extra run.
Top comments (0)