APE: let a model write, score, and pick your prompt for you

Hand-writing a good prompt is guesswork. You try a phrasing, eyeball a handful of outputs, tweak a word, run it again, and stop when it "feels" right. That process is slow, subjective, and it only ever explores the three or four phrasings you happened to think of. Automatic Prompt Engineer (APE) throws that away and treats prompt writing as search: give it examples, let a model generate candidate instructions, score each one objectively, and keep the winner.

🔮 Interactive demo: https://dev48v.infy.uk/prompt/day33-ape.html

The idea (Zhou et al., 2022)

The paper "Large Language Models Are Human-Level Prompt Engineers" proposes using an LLM to generate, score, and select its own instructions. The elegant part is that the model plays two roles at once. As the proposer, it reads a few input→output examples and drafts candidate instructions. As the executor, it runs each candidate to measure how well it actually works. No fine-tuning, no gradients — just sampling instructions and evaluating them. On instruction-induction benchmarks, the prompts APE discovered matched or beat human-written ones.

You don't write the prompt — you provide examples

APE starts from a task defined only by demonstrations. Say the task is "give the opposite of a word":

happy → sad
hot   → cold
fast  → slow
up    → down

Split those examples into two piles: a few demos the proposer will read, and a held-out set it never sees. The held-out split is what keeps the whole thing honest — it measures whether an instruction generalizes, not whether it memorized the demos.

Propose → score → select

The loop has three beats.

1. Propose (forward generation). Show the model the demos and ask, in effect, "what instruction maps these inputs to these outputs?" Sample it several times at a nonzero temperature and you get a diverse pool:

• "Write the antonym of the input word."
• "Give the opposite of the word."
• "Reverse the meaning of the word."
• "Output a word related to the input word."

You are asking the model to reverse-engineer the task from examples.

2. Score. This is the objective function. Run each candidate on every held-out input and count exact matches — execution accuracy:

async function score(instruction, heldout) {
  let correct = 0;
  for (const ex of heldout) {
    const pred = await llm(`${instruction}\nInput: ${ex.in}\nOutput:`);
    if (pred.trim().toLowerCase() === ex.out) correct++;
  }
  return correct / heldout.length;   // 0..1, on UNSEEN examples
}

(The log-probability the model assigns to the correct output is the other classic scorer — smoother and cheaper.) There is no human judgement in the loop.

3. Select. Sort by score, take the top. "Write the antonym of the input word" lands at 83% and wins; "output a word related to the input" collapses to 17%. The chosen prompt is the one that demonstrably works best, not the one that read most elegantly.

Resample near the winner

One round rarely hits the ceiling. APE adds an iterative Monte-Carlo step: take the current best instruction and ask the model for semantically similar variations, then score those too and keep the new champion. This is local search around what already works — and it routinely finds a sharper phrasing the broad first round never proposed:

round 1 winner:  "Write the antonym of the input word."          83%
resample near it → "Write the single-word antonym (exact
                    opposite) of the input word."                100%

Adding "single word" and "exact opposite" disambiguated the one case round 1 got wrong (is the opposite of hard "soft" or "easy"?). Because you always keep the incumbent, the best score never drops across rounds.

Why it beats hand-writing

Two reasons. Coverage — it tries dozens of phrasings and tests all of them, while a person tries a few. Objectivity — it selects on measured accuracy, so it is immune to the intuition traps that fool us; the phrasing that sounds authoritative is often not the one that scores best. The famous result: an APE-discovered prompt ("Let's work this out step by step to be sure we have the right answer") beat the canonical human "Let's think step by step" on several benchmarks.

The cost, and the limits

APE is not free. Each round costs candidates × held-out examples model calls, times the rounds — hundreds of calls to tune one prompt. The saving grace: you pay it offline, once, then ship the single winning instruction and reuse it forever. And it only works when its two ingredients exist: a labelled dev set and a computable scorer (exact match, F1, a unit test, or an LLM-judge). Open-ended tasks with no ground truth have nothing to optimize. Also watch overfitting a tiny held-out set, and remember the found prompt is tuned to one model — re-run APE when you switch.

APE is the ancestor of today's gradient-free prompt optimizers: DSPy's BootstrapFewShot and MIPRO, OPRO, evolutionary prompt search. They all industrialize the same loop — propose, score, select. Once you see prompting as search with a scorer, you stop typing prompts and start optimizing them.

Step through both rounds on the demo: https://dev48v.infy.uk/prompt/day33-ape.html