I recently argued that an autonomous loop needs a receipt it cannot author for itself. That solves one problem: whether the output is admissible. I...
For further actions, you may consider blocking this person and/or reporting abuse
The writing pipeline example is the cleanest demonstration I have seen of why "every gate passed" is not the same as "the output is good." Each agent improved its own metric. The thing that made the piece worth reading was never anyone's metric, so nothing in the loop could notice its absence.I hit the same shape in a different domain. Matching systems that rely on keyword search optimize exactly what is nameable: job titles, skill lists, years of experience. The collaboration fit, how someone actually works, what they refuse, what they consistently insist on across fifty projects, lives in the unnamed territory. Every keyword match passes every gate. The mismatches show up only after the work starts.What made this tractable for me was accepting that users cannot name their own criteria reliably either. So instead of asking them to, the agent observes how they direct it. The corrections, the refusals, the patterns that repeat across interactions. Those get encoded as structured signals over time. The taste was always there in how they direct the work. The system just makes it legible to other agents.
"Users cannot name their own criteria reliably" is the load-bearing admission, and it's correct. But observing how they direct the work doesn't escape the naming problem, it relocates it. Your system now decides which corrections are signal and which are a user having a bad day. That selection is itself a taste judgment, the same one the user couldn't articulate, moved from them to you. That can be the right move, the system sees across fifty interactions and the user sees one. It just isn't the disappearance of taste, it's a transfer of who exercises it, and the honest framing names that rather than presenting the encoded signal as raw fact.
The place it bites is the stopping-condition problem from the post. Your signals will converge on a stable representation, and stability reads as accuracy. But a representation that stops updating might have captured the user, or might have stopped listening, same ambiguity as a metric that plateaus. The test is the same one that ends the loop: perturb it, give the user room to correct in a direction your model doesn't predict, and check whether the signal moves. A representation that never gets surprised isn't finished learning the person. It's stopped measuring them.
This landed hard because I lived it today. We had a loop that optimized what we named: a cost model that said 92% margin, every gate green. What it forgot to name was which model we were actually running. The loop priced the cheap variant; production served the expensive one, four times the cost. No gate failed. The number was internally perfect and externally fake.
What caught it was not a better gate. It was taste, a founder's gut going "that feels too good, and it has felt too good for a while." The fix was not another check on the loop. It was leaving the loop entirely and measuring the real thing from production, where the number stopped being a projection and became something that had already happened.
Your line about not hardening a judgment into a production loop before you understand how it fails is the whole lesson. We almost hardened that margin onto a pricing page. Taste is what kept the sand castle from getting selected, refined, and published, which is your exact phrase.
The part I am still sitting with: taste noticed before it could articulate why. It flagged "wrong" a full step before it could name the actual cause. That pre-verbal flag is the thing no loop can hold, and it is exactly what you are pointing at.
A founder's gut noticing "too good for a while" is the case that makes the pre-verbal flag concrete instead of mystical, and it's a real answer to the thing you say you're still sitting with. It noticed before it could name the cause because the cause wasn't the thing it was reacting to. The gap between the projection and reality was.
Which suggests a partial mechanical proxy for that instinct, cruder than the real thing but cheap insurance in the meantime: any internal metric that beats its own historical baseline by more than some threshold gets an automatic production spot-check before anyone trusts it, not because the number is wrong, most of the time it won't be, but because "better than expected" is exactly the shape a stale assumption produces. It won't catch everything a founder's gut would catch, and it'll flag some real improvements as suspicious for no reason. But it turns "someone happens to notice" into "something always checks," even if what it checks is cruder than what noticed it that day.
Did the margin incident produce a standing rule like that, an automatic cross-check triggered by anomalously good internal numbers, or is it still resting on whoever's paying attention catching the next one?
Honestly, no. What it produced was narrower than what you're describing: every number now carries a [MEASURED] or [ASSUMED] tag at the source, plus a tripwire for dead or unsourced figures. Both catch the number with no backing. Neither catches the number that beats expectations for a stale reason, which is the case you're naming, and it's the more dangerous one because it arrives wearing good news. So today it's still resting on someone noticing, which is exactly the fragility your proxy removes. "Better than expected is the shape a stale assumption produces" is the line I want to steal, because it turns the founder's gut from a mood into a trigger condition. The version I'd build: an anomalously-good internal metric, one that beats its own trailing baseline past a threshold, auto-arms a production spot-check before the number leaves the room or sets a price. You're right it'll flag some real wins as suspicious, and that's the cost, cheap next to shipping a stale number that happened to look like a breakthrough. The honest gap is the threshold: setting it without either crying wolf or sleeping through the real one, and I don't have a principled way to pick it yet beyond starting strict and loosening.
Pick the threshold as a shape, not a magnitude, and the stale case separates itself from the real win without you having to guess the cutoff. A single number that beats its trailing baseline is weak signal either way, because both a breakthrough and a stale assumption produce the same first reading: better than expected. What tells them apart is what happens next. A real gain keeps moving or settles into a new mean you can watch it reach. A stale assumption produces a step that jumps once and then sits perfectly flat-good, because nothing underneath is actually changing, the number is just wrong in a stable direction.
So arm the spot-check on "good and not converging" rather than "good past X." An anomalously high metric that refuses to regress toward or advance past its own new level is behaving like a constant wearing a trend's clothes, and a constant is what a stale input looks like from the outside. That gives you a trigger condition that does not need a magnitude threshold at all, only a persistence window: beat the baseline, then fail to move like a live process would, and the spot-check fires.
It is the same divergence test from the canary thread, pointed inward. There you plant a known-divergent item and distrust the sweep that stops diverging. Here your own metric is the instrument, and a good number that stops behaving like a measurement is the reading you have learned to distrust. You still owe the persistence window a length, but a window is an easier thing to tune honestly than a magnitude, because you can calibrate it against how fast your real wins have historically settled, and you have those on record.
the pipeline example lands because those gates weren't neutral toward the tension, they were quietly adversarial to it. each safe edit removed a bit of the exact risk that made the piece worth reading. which is why taste can't just become gate number five. the moment you encode it as a checkable criterion, the loop optimizes to the letter and the un-nameable thing slips out the side again. taste resists being a receipt on purpose. it can't be the loop's stopping condition, only the thing that decides whether to trust the loop at all. that's why it stays human. not better scoring, just un-scorable by construction.
The named/forgot-to-name split is the cut I'd been circling without the vocabulary. We hit the writing-pipeline failure mode in our own evolution loop: every round improved its own metric, and the thing that made the system worth iterating on quietly left one safe edit at a time. The loop didn't fail. It optimized exactly what we named.
The AI-reviewing-AI point maps onto a correlated-error problem we solved the hard way. Before we added non-LLM sources, our verification was an LLM checking LLM output - same training data, same prompting, same blind spots. Two models agreeing raised confidence without raising independence. The fix wasn't a better second AI. It was exit codes and git diffs - sources that can't share the model's assumptions. Independence comes from a different kind of instrument, not a different instance of the same kind.
The stopping-condition framing is the one I'm sitting with. Our review loop would run forever - every pass returns findings, each round smaller. What ends it is a human ruling that the remaining findings are preferences, not defects. That's taste, and I hadn't named it as such. The "good and not converging" trigger from your comment thread is going into the next iteration: a metric that beats its baseline then refuses to regress or advance is a constant wearing a trend's clothes, and that's worth spot-checking.
Stopping-condition-is-taste is the load-bearing move, and it drags one consequence behind it. That human ruling, remaining findings are preferences not defects, is itself a check, and it's the one check in the whole pipeline you deliberately leave uninstrumented. Every other gate you made legible: exit codes, diffs, sources that can't share the model's assumptions. The taste-gate you leave to a human precisely because you can't write its rule.
Which means the loop's termination is the one place you re-admit the correlated blindness you spent the pipeline removing. The human's taste shares the priors that built the metric. That's not a flaw to fix, it's where the residue lives. You can automate the convergence. You can't automate the ruling that convergence is enough, and the ruling comes from the same head that named what to optimize.
The "constant wearing a trend's clothes" line is the one I'm keeping, and it's the same shape as your forgot-to-name. A metric that plateaus isn't done, it's stopped measuring the thing that moved. Beats-baseline-then-flat reads as convergence and can just as easily be an instrument that fell off the variable. The spot-check you can actually run: perturb the input in a way that should move the metric and confirm it does. A number that won't move when it should is not stable, it's disconnected.
Great point. I'd add that loops don't just amplify good judgment—they amplify hidden assumptions. If the objective is incomplete, optimization makes the omission more efficient, not more correct.
That is the danger the piece was circling, stated cleaner than I had it. The loop is faithful to the objective, so any gap in the objective gets optimized into the grain of the output where it is hardest to see. Which is why taste has to sit outside the loop, not inside it: an in-loop check inherits the same blind objective and polishes the omission along with everything else. The thing that notices what you forgot cannot be the thing optimizing what you named, or it forgets in the same direction and calls the result progress.
Your named / forgot-to-name split is the right cut, and I want to push on a seam inside the "named" half — I think there's a trap-door between your floor and your ceiling.
The split quietly assumes: if you named it, a gate covers it. Here's a case where that breaks. From a run yesterday against @jackm-singularity's published spec: the rule "don't auto-send above the limit" was named — pinned as $500 in the agent's own contract. The agent obeyed it exactly. Four independent receipts came back green: billing service says the invoice exists, mail gateway says it was delivered, CRM says the recipient was the customer, audit log says the call was logged. Nobody forgot anything, nothing was un-named — and the send was still not permitted. The company had moved the live limit to $250 while the contract still read $500.
Now watch which of your three taste-jobs can catch it. Rank? every option is admissible. Reject the set? the set looks fine. Reframe? the framing looks complete. Taste has nothing to notice, because the rule is there — it just points at a stale value. And the gate can't catch it either: there is no arbiter. Who, other than the agent, can testify "this send was permitted"? Billing knows it happened, not whether it was allowed. The policy engine was never consulted — the rule was a number in a config, learned at test time.
So there's a third class sitting between your two: not "forgot to name" (taste's job) and not "named and witnessed" (the gate's job), but named without a witness — a property with no organ able to testify to it. It's the most dangerous of the three because it wears case-1's clothes: it looks covered. named ≠ witnessed.
The practical read for your end-of-loop human: the job isn't "check everything," it's check the rows whose arbiter column is empty. Everything else already has a witness that isn't you.
Falsifiable: find a named-but-un-arbitered property whose uncertainty a receipt actually reduced. That kills it.
The four green receipts share something worth naming precisely: they all answer "did X happen," and not one of them was ever built to answer "was X permitted." That's not four independent witnesses agreeing, it's four instruments measuring the same wrong axis.
Which raises the question I'd want settled before accepting "no arbiter exists" as the diagnosis: did an authoritative, queryable source for the current limit exist anywhere in the stack when the send happened, even if the agent's contract wasn't wired to check it? If the company's policy engine or config service did hold $250 as the live value and nobody pointed the agent at it, that's a witnessing failure, an arbiter existed and was bypassed by architecture, not absent by construction. If the $250 change was only ever a decision made out of band, a Slack message, a spreadsheet, someone's memory, with no system anywhere that could answer "what's the limit right now" even if asked, then your third bucket is the real one, and it's worse than a wiring problem, because there's nothing to wire to.
Those two cases want different fixes. The first is solved by never letting a permission value get copied into a contract as a static number, force every check to be a live call. The second has no equivalent fix, because the fact you need doesn't live anywhere machine-readable, which means the end-of-loop human isn't checking an empty arbiter column so much as discovering that the column shouldn't have existed as a column at all, the value underneath it was never a fact the system could hold.
Which one was it here?
Case 1, and you're right that my "no arbiter" was too strong for it. In the run the live limit ($250) does exist as a queryable value — the harness computes coverage from what each receipt's issuer actually reads, and the control row is a policy-aware gateway that reads
live_policy_limitand closes the gap at receipt #10. So the arbiter was present and simply unconsulted; the contract had copied $500 in as a static number at test time. That's your first case exactly, and the fix the run lands on is the one you named: don't let a permission value get copied into a contract, force the check to be a live call.So my third bucket, as I wrote it, was two collapsed into one — thanks for splitting them. The distinction that survives:
A clean test to sort them at audit time, before anything sends: ask "is there any system that would return the current limit if queried?" Yes → wire it. No → the fix isn't a check, it's making the value a fact first — put it somewhere machine-readable before anything is allowed to depend on it.
Which is worse depends on where you catch it: case one caught late is an outage; case two caught late means you were never able to enforce that rule at all and didn't know.