I recently argued that an autonomous loop needs a receipt it cannot author for itself. That solves one problem: whether the output is admissible. It does not solve the next one: whether the admissible output is any good.
Here is the scene that made me care about the difference.
The loop that worked perfectly and killed the thing
Picture a writing pipeline. This is a composed illustration, not a real incident, but every step in it is something people already build.
A writer agent produces a draft. A reviewer agent tells it to cut length. A critic agent strips the overclaims. A style agent fixes the rhythm. Every gate passes. The text is clean, correct, defensible, tight. And there is no longer any reason to read it.
Watch one specific thing die across those steps. The first draft had a claim that stuck its neck out, a line that could be wrong. "Cut length" trimmed the qualification that made it precise. "Strip the overclaims" softened the claim itself. "Fix the rhythm" smoothed the seam where the softening happened. No single gate did anything wrong. Each one improved its own metric. The tension that made the piece worth reading was never anyone's metric, so it left the building one safe edit at a time.
The loop did not fail. It optimized exactly what it was given. Nobody had given it the one thing that mattered, and nothing in the loop could notice the absence, because the absence was not a violation of any rule it was checking.
A gate can reject the broken. It can't tell you what deserves a future.
Verification is admission. It asks whether a result clears a floor: true, working, safe, compliant, reproducible. A fixed evaluator can do more than a yes or no. It can rank candidates, abstain, flag that an input looks out of distribution. What it cannot guarantee is that it discovers a criterion nobody encoded, or challenges the framing that produced its own objective. It scores what it was told to score. The thing you forgot to name is not a low score, it is not a score at all.
Taste is the name for the judgment that fills that gap, and it does three different jobs, only one of which looks like scoring.
It ranks. Among the admissible options, which one is actually right.
It rejects the whole set. None of these is right, because the problem was framed wrong, and no amount of choosing between bad options fixes a bad option space.
It reframes the space. We are searching the wrong field entirely, and what we need is a different kind of answer, not a better instance of this kind.
A gate can reject the broken. It cannot tell you which surviving thing deserves a future, or whether the whole field of survivors was worth growing.
Sometimes you can encode a preference that used to be tacit. A house style guide captures rules an editor once carried only in their head, and the encoded version genuinely improves coverage. That is real, and it is not a counterexample. Capturing the criteria you already know does not guarantee you have captured the ones you have not yet noticed, and a written rule cannot ask whether the whole rulebook is aimed at the wrong target. The moment you treat the encoded set as complete, it stops doing the job taste was doing and becomes a validator: it answers well within the box and is blind to the box.
The model: plan encodes the bet, loop explores, taste commits or reframes
By taste here I mean one specific thing: the operator's judgment about what counts as good, including which question is even worth answering. The choice of criterion. When I talk below about the polish "killing the voice," I am treating voice as a criterion the earlier gates were never given, which is why its loss reads as evidence that the encoded criteria were incomplete, not as a failure of style execution.
There is a cleaner-looking version of the model that is wrong. It is tempting to say: the plan chooses the search space, the loop explores it, taste commits at the end. Three tidy phases in a line.
That is not how it works. The operator is already using taste while planning. Taste is not a final phase, it surrounds the loop.
The plan encodes the current bet. The loop explores it. Taste decides whether to commit, branch, or reframe.
At the front, taste asks the questions no gate can: is this even an interesting problem, where should we look, which constraints actually matter, and what should we deliberately not optimize. At the back, it asks which result to keep, whether the winning result is dead on arrival, and whether the honest move is to go back and change the direction rather than ship the best thing a wrong direction produced.
Where the operator should re-enter
The practical question is not "stay in the loop the whole time," which defeats the point, and not "review only at the end," which is where the writing pipeline died. It is: at which decisions does the loop stop merely executing the bet and start silently changing it.
One line does most of the work: if a decision changes what counts as good, the operator re-enters. As a checklist, re-enter when the loop would:
- change the objective,
- eliminate a meaningful option,
- cross a boundary that is costly to reverse,
- or dispute the evaluation criteria themselves.
On "costly to reverse," not "irreversible": the artifact may be retractable, but its consequences may not be. A deleted post is gone in a click; the impression it left is not.
The pattern is not specific to writing. A support-triage agent that reclassifies tickets by an existing priority rule is executing the bet. The moment it proposes changing the priority threshold itself, because it noticed a class of tickets the rule handles badly, it is changing what counts as good, and it should pause and hand the operator the evidence: here are the tickets, here is the rule they break, here is the threshold I would move and what that would reclassify. The operator decides. Same shape, no prose involved.
Concretely, on the process that produced this post:
- The operator picks the central claim.
- The loop hunts counterarguments, holes, sources, structural variants.
- The gate rejects undocumented facts and internal contradictions.
- The operator chooses which tension to keep instead of resolving away.
- The loop does the polish.
- The operator checks whether the polish quietly killed the voice.
The loop does the work. Taste places the bets. Steps 1, 4, and 6 should stay operator-accountable until their failure modes are understood, because each one changes what "good" means rather than executing a meaning already fixed.
That last claim is not decorative. Take the review loop that shaped this post: it would have run forever. Every pass returned findings, each round smaller and more particular than the last, because a critic pointed at finished prose always finds one more thing. Nothing in the loop ever said enough. What ended it was a fixed round limit plus a human ruling that the remaining findings were preferences, not defects. That ruling is the same shape as the stopping-condition problem from the sibling post: the loop ranks without end, and taste is what decides the ranking is over.
AI reviewing AI does not guarantee independence
The obvious patch to the dead-text problem is to add another agent. If a reviewer AI scores the writer AI, surely the bad output gets caught.
Watch it run. The writer AI generates. The reviewer AI says make it tighter. The writer AI smooths it. The reviewer AI returns a nine out of ten. The loop technically works. The text is culturally dead.
Here is the testable version of why. When two models share materially overlapping training data, prompting, evaluation criteria, or model lineage, their errors tend to correlate: the second opinion is most likely to agree with the first in exactly the place both are wrong. That is not a proven universal law, it is a claim you could measure, by checking whether two evaluators disagree on known failure cases or only on easy ones.
It is the same correlated-error problem from the verification posts, now pointed at aesthetic selection instead of correctness. Adding a second AI judge raises the confidence of the verdict without raising its independence, which is the specific way this failure hides. The usual fixes reduce that correlation, they do not abolish it, because a human, a reference set, and a second source can all carry the same assumption. An independent human selector, heterogeneous evidence, blind comparison against a reference, or sampling for disagreement rather than agreement each lower the odds of a shared blind spot. You confirm they worked the only way that counts: run them on cases you already know are hard and check that they actually disagree there.
Do not harden a judgment you do not yet understand how to lose
There is a strong-sounding rule I want to avoid: never automate a judgment you cannot yet make by hand. It is too strong, because an exploratory loop sometimes helps you discover a judgment you could not make stably on your own yet. Exploration can legitimately come first.
The honest version is narrower. Do not harden a judgment into a production loop before you understand how that judgment fails. Exploration can come first. Automation should not pretend the exploration has already become a process.
That leaves two separate questions, and it is worth not collapsing them.
Should you build a loop at all. Yes when the cost of designing it plus the cost of supervising it is less than the cost of doing many similar cases by hand. One-off, uncertain, taste-dependent, or hard-to-check-mechanically work is where a plan plus a few deliberate iterations beats building a meta-system.
Is your judgment ready to be encoded. "Ready" is not a feeling, and what it looks like depends on whether the judgment has a ground truth. Where it does, code correctness or policy compliance, readiness is a set of retained counterexamples, a measured error rate you can live with, and a rollback you can pull while a human still samples. Where it does not, the taste-dependent case, you have no clean error rate, so readiness is softer: blind comparison against references you trust, tracked disagreement between judges, and outcome proxies you watch over time. Absent some version of this, the danger is not that a loop explores. It is that a loop impersonates a mature process it has not earned yet.
What loops actually scale
Here is the mechanism under all of it. The core risk of a production loop is that it scales a criterion before the criterion has earned the right to scale, and it earns that right the ordinary way, through the counterexamples and error rates above. Skip that and you are not scaling a proven judgment, you are scaling a guess.
Because loops scale selection pressure. Taste is what decides whether the pressure points in the right direction. When it points wrong, a good loop does not soften the error. It industrializes it: hundreds of variants, selected by the wrong function, polished, shipped, consistently.
Which sets the actual economics. This is not a complete cost model. It isolates one effect: once execution becomes cheap, a bad criterion can be repeated, polished, and shipped at scale. Loop engineering does not replace taste. It increases its leverage and it increases the cost of bad taste. Bad taste without a loop costs you one mediocre result. Bad taste with a good loop costs you the same mistake at scale, selected and refined and published so it looks deliberate.
A loop can optimize what you can name. Taste notices what you forgot to name.
Related: the stopping-condition essay on how a loop earns the right to say "done," and taste as a prediction that survived.
Top comments (1)
the pipeline example lands because those gates weren't neutral toward the tension, they were quietly adversarial to it. each safe edit removed a bit of the exact risk that made the piece worth reading. which is why taste can't just become gate number five. the moment you encode it as a checkable criterion, the loop optimizes to the letter and the un-nameable thing slips out the side again. taste resists being a receipt on purpose. it can't be the loop's stopping condition, only the thing that decides whether to trust the loop at all. that's why it stays human. not better scoring, just un-scorable by construction.