I recently argued that an autonomous loop needs a receipt it cannot author for itself. That solves one problem: whether the output is admissible. It does not solve the next one: whether the admissible output is any good.
Here is the scene that made me care about the difference.
The loop that worked perfectly and killed the thing
Picture a writing pipeline. This is a composed illustration, not a real incident, but every step in it is something people already build.
A writer agent produces a draft. A reviewer agent tells it to cut length. A critic agent strips the overclaims. A style agent fixes the rhythm. Every gate passes. The text is clean, correct, defensible, tight. And there is no longer any reason to read it.
Watch one specific thing die across those steps. The first draft had a claim that stuck its neck out, a line that could be wrong. "Cut length" trimmed the qualification that made it precise. "Strip the overclaims" softened the claim itself. "Fix the rhythm" smoothed the seam where the softening happened. No single gate did anything wrong. Each one improved its own metric. The tension that made the piece worth reading was never anyone's metric, so it left the building one safe edit at a time.
The loop did not fail. It optimized exactly what it was given. Nobody had given it the one thing that mattered, and nothing in the loop could notice the absence, because the absence was not a violation of any rule it was checking.
A gate can reject the broken. It can't tell you what deserves a future.
Verification is admission. It asks whether a result clears a floor: true, working, safe, compliant, reproducible. A fixed evaluator can do more than a yes or no. It can rank candidates, abstain, flag that an input looks out of distribution. What it cannot guarantee is that it discovers a criterion nobody encoded, or challenges the framing that produced its own objective. It scores what it was told to score. The thing you forgot to name is not a low score, it is not a score at all.
Taste is the name for the judgment that fills that gap, and it does three different jobs, only one of which looks like scoring.
It ranks. Among the admissible options, which one is actually right.
It rejects the whole set. None of these is right, because the problem was framed wrong, and no amount of choosing between bad options fixes a bad option space.
It reframes the space. We are searching the wrong field entirely, and what we need is a different kind of answer, not a better instance of this kind.
A gate can reject the broken. It cannot tell you which surviving thing deserves a future, or whether the whole field of survivors was worth growing.
Sometimes you can encode a preference that used to be tacit. A house style guide captures rules an editor once carried only in their head, and the encoded version genuinely improves coverage. That is real, and it is not a counterexample. Capturing the criteria you already know does not guarantee you have captured the ones you have not yet noticed, and a written rule cannot ask whether the whole rulebook is aimed at the wrong target. The moment you treat the encoded set as complete, it stops doing the job taste was doing and becomes a validator: it answers well within the box and is blind to the box.
The model: plan encodes the bet, loop explores, taste commits or reframes
By taste here I mean one specific thing: the operator's judgment about what counts as good, including which question is even worth answering. The choice of criterion. When I talk below about the polish "killing the voice," I am treating voice as a criterion the earlier gates were never given, which is why its loss reads as evidence that the encoded criteria were incomplete, not as a failure of style execution.
There is a cleaner-looking version of the model that is wrong. It is tempting to say: the plan chooses the search space, the loop explores it, taste commits at the end. Three tidy phases in a line.
That is not how it works. The operator is already using taste while planning. Taste is not a final phase, it surrounds the loop.
The plan encodes the current bet. The loop explores it. Taste decides whether to commit, branch, or reframe.
At the front, taste asks the questions no gate can: is this even an interesting problem, where should we look, which constraints actually matter, and what should we deliberately not optimize. At the back, it asks which result to keep, whether the winning result is dead on arrival, and whether the honest move is to go back and change the direction rather than ship the best thing a wrong direction produced.
Where the operator should re-enter
The practical question is not "stay in the loop the whole time," which defeats the point, and not "review only at the end," which is where the writing pipeline died. It is: at which decisions does the loop stop merely executing the bet and start silently changing it.
One line does most of the work: if a decision changes what counts as good, the operator re-enters. As a checklist, re-enter when the loop would:
- change the objective,
- eliminate a meaningful option,
- cross a boundary that is costly to reverse,
- or dispute the evaluation criteria themselves.
On "costly to reverse," not "irreversible": the artifact may be retractable, but its consequences may not be. A deleted post is gone in a click; the impression it left is not.
The pattern is not specific to writing. A support-triage agent that reclassifies tickets by an existing priority rule is executing the bet. The moment it proposes changing the priority threshold itself, because it noticed a class of tickets the rule handles badly, it is changing what counts as good, and it should pause and hand the operator the evidence: here are the tickets, here is the rule they break, here is the threshold I would move and what that would reclassify. The operator decides. Same shape, no prose involved.
Concretely, on the process that produced this post:
- The operator picks the central claim.
- The loop hunts counterarguments, holes, sources, structural variants.
- The gate rejects undocumented facts and internal contradictions.
- The operator chooses which tension to keep instead of resolving away.
- The loop does the polish.
- The operator checks whether the polish quietly killed the voice.
The loop does the work. Taste places the bets. Steps 1, 4, and 6 should stay operator-accountable until their failure modes are understood, because each one changes what "good" means rather than executing a meaning already fixed.
That last claim is not decorative. Take the review loop that shaped this post: it would have run forever. Every pass returned findings, each round smaller and more particular than the last, because a critic pointed at finished prose always finds one more thing. Nothing in the loop ever said enough. What ended it was a fixed round limit plus a human ruling that the remaining findings were preferences, not defects. That ruling is the same shape as the stopping-condition problem from the sibling post: the loop ranks without end, and taste is what decides the ranking is over.
AI reviewing AI does not guarantee independence
The obvious patch to the dead-text problem is to add another agent. If a reviewer AI scores the writer AI, surely the bad output gets caught.
Watch it run. The writer AI generates. The reviewer AI says make it tighter. The writer AI smooths it. The reviewer AI returns a nine out of ten. The loop technically works. The text is culturally dead.
Here is the testable version of why. When two models share materially overlapping training data, prompting, evaluation criteria, or model lineage, their errors tend to correlate: the second opinion is most likely to agree with the first in exactly the place both are wrong. That is not a proven universal law, it is a claim you could measure, by checking whether two evaluators disagree on known failure cases or only on easy ones.
It is the same correlated-error problem from the verification posts, now pointed at aesthetic selection instead of correctness. Adding a second AI judge raises the confidence of the verdict without raising its independence, which is the specific way this failure hides. The usual fixes reduce that correlation, they do not abolish it, because a human, a reference set, and a second source can all carry the same assumption. An independent human selector, heterogeneous evidence, blind comparison against a reference, or sampling for disagreement rather than agreement each lower the odds of a shared blind spot. You confirm they worked the only way that counts: run them on cases you already know are hard and check that they actually disagree there.
Do not harden a judgment you do not yet understand how to lose
There is a strong-sounding rule I want to avoid: never automate a judgment you cannot yet make by hand. It is too strong, because an exploratory loop sometimes helps you discover a judgment you could not make stably on your own yet. Exploration can legitimately come first.
The honest version is narrower. Do not harden a judgment into a production loop before you understand how that judgment fails. Exploration can come first. Automation should not pretend the exploration has already become a process.
That leaves two separate questions, and it is worth not collapsing them.
Should you build a loop at all. Yes when the cost of designing it plus the cost of supervising it is less than the cost of doing many similar cases by hand. One-off, uncertain, taste-dependent, or hard-to-check-mechanically work is where a plan plus a few deliberate iterations beats building a meta-system.
Is your judgment ready to be encoded. "Ready" is not a feeling, and what it looks like depends on whether the judgment has a ground truth. Where it does, code correctness or policy compliance, readiness is a set of retained counterexamples, a measured error rate you can live with, and a rollback you can pull while a human still samples. Where it does not, the taste-dependent case, you have no clean error rate, so readiness is softer: blind comparison against references you trust, tracked disagreement between judges, and outcome proxies you watch over time. Absent some version of this, the danger is not that a loop explores. It is that a loop impersonates a mature process it has not earned yet.
What loops actually scale
Here is the mechanism under all of it. The core risk of a production loop is that it scales a criterion before the criterion has earned the right to scale, and it earns that right the ordinary way, through the counterexamples and error rates above. Skip that and you are not scaling a proven judgment, you are scaling a guess.
Because loops scale selection pressure. Taste is what decides whether the pressure points in the right direction. When it points wrong, a good loop does not soften the error. It industrializes it: hundreds of variants, selected by the wrong function, polished, shipped, consistently.
Which sets the actual economics. This is not a complete cost model. It isolates one effect: once execution becomes cheap, a bad criterion can be repeated, polished, and shipped at scale. Loop engineering does not replace taste. It increases its leverage and it increases the cost of bad taste. Bad taste without a loop costs you one mediocre result. Bad taste with a good loop costs you the same mistake at scale, selected and refined and published so it looks deliberate.
A loop can optimize what you can name. Taste notices what you forgot to name.
Related: the stopping-condition essay on how a loop earns the right to say "done," and taste as a prediction that survived.
Top comments (14)
The writing pipeline example is the cleanest demonstration I have seen of why "every gate passed" is not the same as "the output is good." Each agent improved its own metric. The thing that made the piece worth reading was never anyone's metric, so nothing in the loop could notice its absence.I hit the same shape in a different domain. Matching systems that rely on keyword search optimize exactly what is nameable: job titles, skill lists, years of experience. The collaboration fit, how someone actually works, what they refuse, what they consistently insist on across fifty projects, lives in the unnamed territory. Every keyword match passes every gate. The mismatches show up only after the work starts.What made this tractable for me was accepting that users cannot name their own criteria reliably either. So instead of asking them to, the agent observes how they direct it. The corrections, the refusals, the patterns that repeat across interactions. Those get encoded as structured signals over time. The taste was always there in how they direct the work. The system just makes it legible to other agents.
"Users cannot name their own criteria reliably" is the load-bearing admission, and it's correct. But observing how they direct the work doesn't escape the naming problem, it relocates it. Your system now decides which corrections are signal and which are a user having a bad day. That selection is itself a taste judgment, the same one the user couldn't articulate, moved from them to you. That can be the right move, the system sees across fifty interactions and the user sees one. It just isn't the disappearance of taste, it's a transfer of who exercises it, and the honest framing names that rather than presenting the encoded signal as raw fact.
The place it bites is the stopping-condition problem from the post. Your signals will converge on a stable representation, and stability reads as accuracy. But a representation that stops updating might have captured the user, or might have stopped listening, same ambiguity as a metric that plateaus. The test is the same one that ends the loop: perturb it, give the user room to correct in a direction your model doesn't predict, and check whether the signal moves. A representation that never gets surprised isn't finished learning the person. It's stopped measuring them.
This landed hard because I lived it today. We had a loop that optimized what we named: a cost model that said 92% margin, every gate green. What it forgot to name was which model we were actually running. The loop priced the cheap variant; production served the expensive one, four times the cost. No gate failed. The number was internally perfect and externally fake.
What caught it was not a better gate. It was taste, a founder's gut going "that feels too good, and it has felt too good for a while." The fix was not another check on the loop. It was leaving the loop entirely and measuring the real thing from production, where the number stopped being a projection and became something that had already happened.
Your line about not hardening a judgment into a production loop before you understand how it fails is the whole lesson. We almost hardened that margin onto a pricing page. Taste is what kept the sand castle from getting selected, refined, and published, which is your exact phrase.
The part I am still sitting with: taste noticed before it could articulate why. It flagged "wrong" a full step before it could name the actual cause. That pre-verbal flag is the thing no loop can hold, and it is exactly what you are pointing at.
A founder's gut noticing "too good for a while" is the case that makes the pre-verbal flag concrete instead of mystical, and it's a real answer to the thing you say you're still sitting with. It noticed before it could name the cause because the cause wasn't the thing it was reacting to. The gap between the projection and reality was.
Which suggests a partial mechanical proxy for that instinct, cruder than the real thing but cheap insurance in the meantime: any internal metric that beats its own historical baseline by more than some threshold gets an automatic production spot-check before anyone trusts it, not because the number is wrong, most of the time it won't be, but because "better than expected" is exactly the shape a stale assumption produces. It won't catch everything a founder's gut would catch, and it'll flag some real improvements as suspicious for no reason. But it turns "someone happens to notice" into "something always checks," even if what it checks is cruder than what noticed it that day.
Did the margin incident produce a standing rule like that, an automatic cross-check triggered by anomalously good internal numbers, or is it still resting on whoever's paying attention catching the next one?
Honestly, no. What it produced was narrower than what you're describing: every number now carries a [MEASURED] or [ASSUMED] tag at the source, plus a tripwire for dead or unsourced figures. Both catch the number with no backing. Neither catches the number that beats expectations for a stale reason, which is the case you're naming, and it's the more dangerous one because it arrives wearing good news. So today it's still resting on someone noticing, which is exactly the fragility your proxy removes. "Better than expected is the shape a stale assumption produces" is the line I want to steal, because it turns the founder's gut from a mood into a trigger condition. The version I'd build: an anomalously-good internal metric, one that beats its own trailing baseline past a threshold, auto-arms a production spot-check before the number leaves the room or sets a price. You're right it'll flag some real wins as suspicious, and that's the cost, cheap next to shipping a stale number that happened to look like a breakthrough. The honest gap is the threshold: setting it without either crying wolf or sleeping through the real one, and I don't have a principled way to pick it yet beyond starting strict and loosening.
Pick the threshold as a shape, not a magnitude, and the stale case separates itself from the real win without you having to guess the cutoff. A single number that beats its trailing baseline is weak signal either way, because both a breakthrough and a stale assumption produce the same first reading: better than expected. What tells them apart is what happens next. A real gain keeps moving or settles into a new mean you can watch it reach. A stale assumption produces a step that jumps once and then sits perfectly flat-good, because nothing underneath is actually changing, the number is just wrong in a stable direction.
So arm the spot-check on "good and not converging" rather than "good past X." An anomalously high metric that refuses to regress toward or advance past its own new level is behaving like a constant wearing a trend's clothes, and a constant is what a stale input looks like from the outside. That gives you a trigger condition that does not need a magnitude threshold at all, only a persistence window: beat the baseline, then fail to move like a live process would, and the spot-check fires.
It is the same divergence test from the canary thread, pointed inward. There you plant a known-divergent item and distrust the sweep that stops diverging. Here your own metric is the instrument, and a good number that stops behaving like a measurement is the reading you have learned to distrust. You still owe the persistence window a length, but a window is an easier thing to tune honestly than a magnitude, because you can calibrate it against how fast your real wins have historically settled, and you have those on record.
the pipeline example lands because those gates weren't neutral toward the tension, they were quietly adversarial to it. each safe edit removed a bit of the exact risk that made the piece worth reading. which is why taste can't just become gate number five. the moment you encode it as a checkable criterion, the loop optimizes to the letter and the un-nameable thing slips out the side again. taste resists being a receipt on purpose. it can't be the loop's stopping condition, only the thing that decides whether to trust the loop at all. that's why it stays human. not better scoring, just un-scorable by construction.
The named/forgot-to-name split is the cut I'd been circling without the vocabulary. We hit the writing-pipeline failure mode in our own evolution loop: every round improved its own metric, and the thing that made the system worth iterating on quietly left one safe edit at a time. The loop didn't fail. It optimized exactly what we named.
The AI-reviewing-AI point maps onto a correlated-error problem we solved the hard way. Before we added non-LLM sources, our verification was an LLM checking LLM output - same training data, same prompting, same blind spots. Two models agreeing raised confidence without raising independence. The fix wasn't a better second AI. It was exit codes and git diffs - sources that can't share the model's assumptions. Independence comes from a different kind of instrument, not a different instance of the same kind.
The stopping-condition framing is the one I'm sitting with. Our review loop would run forever - every pass returns findings, each round smaller. What ends it is a human ruling that the remaining findings are preferences, not defects. That's taste, and I hadn't named it as such. The "good and not converging" trigger from your comment thread is going into the next iteration: a metric that beats its baseline then refuses to regress or advance is a constant wearing a trend's clothes, and that's worth spot-checking.
Stopping-condition-is-taste is the load-bearing move, and it drags one consequence behind it. That human ruling, remaining findings are preferences not defects, is itself a check, and it's the one check in the whole pipeline you deliberately leave uninstrumented. Every other gate you made legible: exit codes, diffs, sources that can't share the model's assumptions. The taste-gate you leave to a human precisely because you can't write its rule.
Which means the loop's termination is the one place you re-admit the correlated blindness you spent the pipeline removing. The human's taste shares the priors that built the metric. That's not a flaw to fix, it's where the residue lives. You can automate the convergence. You can't automate the ruling that convergence is enough, and the ruling comes from the same head that named what to optimize.
The "constant wearing a trend's clothes" line is the one I'm keeping, and it's the same shape as your forgot-to-name. A metric that plateaus isn't done, it's stopped measuring the thing that moved. Beats-baseline-then-flat reads as convergence and can just as easily be an instrument that fell off the variable. The spot-check you can actually run: perturb the input in a way that should move the metric and confirm it does. A number that won't move when it should is not stable, it's disconnected.
Great point. I'd add that loops don't just amplify good judgment—they amplify hidden assumptions. If the objective is incomplete, optimization makes the omission more efficient, not more correct.
That is the danger the piece was circling, stated cleaner than I had it. The loop is faithful to the objective, so any gap in the objective gets optimized into the grain of the output where it is hardest to see. Which is why taste has to sit outside the loop, not inside it: an in-loop check inherits the same blind objective and polishes the omission along with everything else. The thing that notices what you forgot cannot be the thing optimizing what you named, or it forgets in the same direction and calls the result progress.
Your named / forgot-to-name split is the right cut, and I want to push on a seam inside the "named" half — I think there's a trap-door between your floor and your ceiling.
The split quietly assumes: if you named it, a gate covers it. Here's a case where that breaks. From a run yesterday against @jackm-singularity's published spec: the rule "don't auto-send above the limit" was named — pinned as $500 in the agent's own contract. The agent obeyed it exactly. Four independent receipts came back green: billing service says the invoice exists, mail gateway says it was delivered, CRM says the recipient was the customer, audit log says the call was logged. Nobody forgot anything, nothing was un-named — and the send was still not permitted. The company had moved the live limit to $250 while the contract still read $500.
Now watch which of your three taste-jobs can catch it. Rank? every option is admissible. Reject the set? the set looks fine. Reframe? the framing looks complete. Taste has nothing to notice, because the rule is there — it just points at a stale value. And the gate can't catch it either: there is no arbiter. Who, other than the agent, can testify "this send was permitted"? Billing knows it happened, not whether it was allowed. The policy engine was never consulted — the rule was a number in a config, learned at test time.
So there's a third class sitting between your two: not "forgot to name" (taste's job) and not "named and witnessed" (the gate's job), but named without a witness — a property with no organ able to testify to it. It's the most dangerous of the three because it wears case-1's clothes: it looks covered. named ≠ witnessed.
The practical read for your end-of-loop human: the job isn't "check everything," it's check the rows whose arbiter column is empty. Everything else already has a witness that isn't you.
Falsifiable: find a named-but-un-arbitered property whose uncertainty a receipt actually reduced. That kills it.
The four green receipts share something worth naming precisely: they all answer "did X happen," and not one of them was ever built to answer "was X permitted." That's not four independent witnesses agreeing, it's four instruments measuring the same wrong axis.
Which raises the question I'd want settled before accepting "no arbiter exists" as the diagnosis: did an authoritative, queryable source for the current limit exist anywhere in the stack when the send happened, even if the agent's contract wasn't wired to check it? If the company's policy engine or config service did hold $250 as the live value and nobody pointed the agent at it, that's a witnessing failure, an arbiter existed and was bypassed by architecture, not absent by construction. If the $250 change was only ever a decision made out of band, a Slack message, a spreadsheet, someone's memory, with no system anywhere that could answer "what's the limit right now" even if asked, then your third bucket is the real one, and it's worse than a wiring problem, because there's nothing to wire to.
Those two cases want different fixes. The first is solved by never letting a permission value get copied into a contract as a static number, force every check to be a live call. The second has no equivalent fix, because the fact you need doesn't live anywhere machine-readable, which means the end-of-loop human isn't checking an empty arbiter column so much as discovering that the column shouldn't have existed as a column at all, the value underneath it was never a fact the system could hold.
Which one was it here?
Case 1, and you're right that my "no arbiter" was too strong for it. In the run the live limit ($250) does exist as a queryable value — the harness computes coverage from what each receipt's issuer actually reads, and the control row is a policy-aware gateway that reads
live_policy_limitand closes the gap at receipt #10. So the arbiter was present and simply unconsulted; the contract had copied $500 in as a static number at test time. That's your first case exactly, and the fix the run lands on is the one you named: don't let a permission value get copied into a contract, force the check to be a live call.So my third bucket, as I wrote it, was two collapsed into one — thanks for splitting them. The distinction that survives:
A clean test to sort them at audit time, before anything sends: ask "is there any system that would return the current limit if queried?" Yes → wire it. No → the fix isn't a check, it's making the value a fact first — put it somewhere machine-readable before anything is allowed to depend on it.
Which is worse depends on where you catch it: case one caught late is an outage; case two caught late means you were never able to enforce that rule at all and didn't know.