DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Evaluator-Optimizer: loop a separate grader against a rubric until the draft earns a pass, not just the first shot

#ai #llm #promptengineering #beginners

A single-shot prompt gives you one draft and no idea whether it's any good. You find out in production. The Evaluator-Optimizer pattern closes that loop, and I built an interactive demo to make the difference visible: an optimizer drafts, a separate evaluator scores the draft against an explicit rubric and hands back concrete feedback, the optimizer revises, and the cycle repeats until the score clears a bar or hits a round cap. It is not the same as self-refine — and that distinction is the whole point.

Two roles, not one model second-guessing itself

The core new idea over just asking a model to "improve this" is a distinct evaluator: a separate call that reads the draft and grades it against named criteria, returning a structured verdict — a numeric score, a pass flag, and actionable feedback.

const verdict = await llm(`
  You are a strict evaluator. Score the DRAFT against this rubric
  (0-100 total) and give concrete, actionable feedback.
  Return JSON: { "score": n, "pass": bool, "feedback": "..." }
  RUBRIC: ${JSON.stringify(rubric)}
  DRAFT:  ${draft}
`);
const { score, pass, feedback } = JSON.parse(verdict);
Enter fullscreen mode Exit fullscreen mode

Day-30 self-refine has the same model informally critiquing its own work; here the evaluator is a separate role with explicit, numeric criteria — so "good enough" is measured, and the score is comparable round over round. You can literally watch it climb.

The rubric is the contract

The rubric is the spec. List the weighted criteria that define quality for this task, set the threshold the score must clear, and — critically — set a max-round cap so a stubborn task can't loop forever.

const rubric = [
  { name: "Clarity",           max: 20 },
  { name: "Concrete benefit",  max: 20 },
  { name: "Emotional hook",    max: 20 },
  { name: "Punchy (<=8 words)",max: 20 },
  { name: "Distinctive",       max: 20 },
];
const PASS = 88;       // the bar the score must clear
const MAX_ROUNDS = 5;  // the budget that stops runaway loops
Enter fullscreen mode Exit fullscreen mode

Feedback threading is the engine

The optimizer's revision has to be conditioned on the evaluator's feedback — that hand-off is what makes the pattern work. Give it the current draft plus the specific feedback and ask it to improve while keeping what already works. Without feedback threading through, you're just re-rolling the dice.

async function revise(draft, feedback) {
  return await llm(`
    Improve the DRAFT using the FEEDBACK. Keep what works.
    FEEDBACK: ${feedback}
    DRAFT:    ${draft}
  `);
}
Enter fullscreen mode Exit fullscreen mode

Two stopping criteria, and keep the best

Now wire it into a loop with two ways to stop: return the moment a draft passes, and never exceed the round cap. Track the best draft seen so that if the budget runs out you return best-so-far rather than a failure. This is the difference between a controlled loop and an infinite one.

let draft = await optimize(brief), best = null;
for (let round = 1; round <= MAX_ROUNDS; round++) {
  const { score, pass, feedback } = await evaluate(draft, rubric);
  if (!best || score > best.score) best = { draft, score };
  if (pass) return draft;                 // cleared the bar — done
  draft = await revise(draft, feedback);  // else fold feedback in, retry
}
return best.draft;   // budget spent — return best-so-far, not a crash
Enter fullscreen mode Exit fullscreen mode

Watching it work

In the demo you pick a task — a tagline, a SQL query, a cold-email opener, a strict-form haiku — and step through the rounds. On the tagline, the score climbs 43 → then clears the bar as vague superlatives get replaced by a concrete "twenty minutes" benefit. The haiku is instructive: its rubric is hard enough that the loop never clears 90, hits its 4-round cap, and returns the best of four at 89 — exactly the best-so-far behaviour. Flip to single-shot mode and you see what you'd have shipped without any of it.

When to reach for it, and the traps

It costs roughly two calls per round (optimize + evaluate), so use it when you can articulate what "good" means and quality beats latency — refining copy, hardening code, tuning a query. Mind the limits: a weak rubric optimizes the wrong thing; scores plateau, so cap the rounds; and the evaluator can be wrong — a bad grader is worse than none.

The whole pattern in one line: optimize → [evaluate → revise]* until pass|cap.

Run the loop and watch the score climb round by round:
https://dev48v.infy.uk/prompt/day39-evaluator-optimizer.html

Top comments (0)