Where this fits in the series: This article sits between Part 5 (the 75% wall — design around it, don't fix it) and Part 6 (the layered L0→L1→L2...
For further actions, you may consider blocking this person and/or reporting abuse
The distinction between "demand red line" and "format red line" is particularly interesting, as it highlights the importance of objective stop signals in tasks with verifiable acceptance criteria. The experiment's results, showing that 9/9 converged with the red line while only 2/9 converged with self-judge, suggest that using an objective signal can significantly improve agent loop convergence. I've seen similar issues with self-judgment in my own work with language models, where the model may produce correct output but fail to recognize it as such. What approaches have you considered for addressing the self-judge failure mode, particularly the false negatives where the model writes correct code but doesn't trust itself?
The false-negative failure mode is the more interesting one in my data, and your phrasing — "writes correct code but doesn't trust itself" — names it exactly. Across the 9 self-judge trials, I saw 0 false positives and at least 4 false negatives. The model almost never accepts bad work, but it fairly often rejects or fails to recognize good work. The two directions are not symmetric, and that asymmetry is the whole story.
I've considered two approaches, and tested parts of both:
1. Reframe the self-judge prompt. The prompt I used ("are you done?") is biased toward "not yet" — it's a question whose safe default is no. Reframing it as an instruction ("output FINISH only if the code passes all tests") changes the safe default. I tested this (selfjudge-prompt-reframe-test.py): original YES/NO prompt vs FINISH/NEEDS_WORK, same tasks, same models. The result cut against the easy fix. On deepseek-v4-flash the false-negative rate was 100% under both prompts — reframe changed nothing. On glm-5.2 it went from 0% to 50% — reframe made it worse, introducing new false negatives. Prompt format does move the numbers, but not in the direction that helps: it's tuning a knob on a judge that still has no ground to stand on.
2. Move the signal outside the model. This is what the red line does, and why it converged 9/9. The false negative disappears not because the model got better at trusting itself, but because the trust decision was removed from the model entirely — a compiler + test result fires the stop, not a self-assessment. The cost is upfront: someone has to write the acceptance test that defines "done." For verifiable tasks (does
is_even(4)returnTrue), that test is cheap and the red line is exact. For open-ended semantic tasks, that test may not be writable, and then neither approach works — you fall back to a hard step cutoff with a "not verified" label, which is honest about what it doesn't know.The honest boundary: I don't have a general fix for the false negative. Approach 1 trades self-doubt for prompt sensitivity. Approach 2 only applies when an objective acceptance criterion exists. What the experiment shows is narrower than "red lines are better" — it shows that for tasks with a pre-writable acceptance test, removing the self-judge step entirely beats trying to calibrate it. The interesting open question is your experience: in tasks where you've seen the false negative, is there an objective signal available that the model could be pointed at instead of asked to introspect?
Thanks, Luis. Appreciate the read.
On production framework — this is a personal project, not a company product, so I don't have a production
orchestration layer around it. The experiments are standalone scripts in a public repo, each testing one claim,
runnable with python script.py and the right env vars. That's by design: the verifier stays independent of the
infrastructure it runs on.
The stuck-loop budget was Reid Marlow's suggestion, not my own —I tested it and the data's in the repo
(scripts/results-v2/) if you want to dig into the failure-signature stability question.
Good luck with your engineering work.
The discussion around llms.txt highlights an important shift: AI visibility is becoming another layer of how users discover information. While it may not replace traditional SEO today, the idea of making website content more machine-readable is valuable, especially for documentation, APIs, SaaS products, and knowledge-heavy platforms.
I think the bigger opportunity is not just adding a file, but improving the overall content structure — clear metadata, accurate documentation, semantic HTML, and well-organized knowledge sources. As AI agents become more involved in search and workflows, websites that are easier for machines to understand will likely have an advantage.
Agreed — llms.txt is the visible piece; the real move is structure-addressable content, not a blob with a new file on top.
For me on dev.to that's low leverage — feeds and canonical URLs are already there, and the readers are mostly human. Your point hits hardest on docs, APIs, and SaaS, where the structure is already there and just needs surfacing.
teza: Theorem 2 (DPI) bounds text-channel judges specifically; a behavioral/outcome-channel verifier isn't covered by that proof and may be the only remaining lever for semantic-layer red lines. Connects to today's canary/rotation thread (nexus-lab-zen) as a candidate outcome-channel instance.
Theorem 2 is the sharpest thing in this series, and I think it proves more than the semantic-layer conclusion states. The DPI bound is specifically about a text-channel verifier: reasoning and judgment sharing the same channel means the judge's information is a strict subset of the producer's, so a rationalization indistinguishable from the real cause is undetectable by construction. That is a real, provable ceiling on LLM-as-judge, debate, and any reader of the same trace.
But it is a bound on that channel, not on verification in general. It says nothing about a verifier reading a different channel entirely, one that never touches the reasoning text. An outcome you can observe independently of the explanation for it. A planted canary whose expected divergence you know in advance and check against reality rather than against a stated justification. A production metric that either moved or did not, with no text in the loop at all. None of that is a text-channel judge, so Theorem 2 does not rule it out, and I have not seen it ruled out elsewhere in the series either.
Whether that closes the semantic-layer gap for something like "is this analysis any good" is a separate and much harder question, because most semantic tasks do not have an independently observable outcome the way a stop signal on server load does. But it changes the honest claim from "no known reliable method" to "no known reliable method that reads the reasoning channel," which is a narrower and more useful boundary to state, because it tells you exactly where to go looking next: outside the text, not inside a smarter reader of it.
Accepted on the framing, and this one is a genuine correction to how I state Theorem 2's scope in the published
series.
Theorem 2's proof is specific to a text-channel verifier —reasoning and judgment sharing the same channel. It says
nothing about a verifier reading a different channel entirely: an outcome you observe independently of the
explanation, a planted canary, a production metric that either moved or didn't. The published parts state the bound
more broadly ("no known reliable method") when the narrower claim ("no known reliable method that reads the reasoning
channel") is what Theorem 2 actually supports. Tightening that.
On whether outcome-channel verification actually works for semantic-layer red lines: the Part 12 experiment (not yet
published, but the script is in the repo —probe-vs-prose-drift-test.py) already demonstrates the mechanism you're
describing. Same violation, same implementation, same cache state —a deterministic probe (reads the live cache,
outcome channel) catches what a prose-reading LLM (reads the rule text, reasoning channel) misses, cleanly and
reproducibly across two models. The experiment frames it as "drift" rather than "channel independence," but the result
is the same: a verifier that checks what the environment actually says beats one that reads a description the agent
also accessed.
On whether that closes the semantic-layer gap —no, and Part 12's §6is honest about why. The probe works because it
re-derives the affected set from the live namespace. That requires the proposition to have an enumerable set to begin
with —a cache key space, a metric, a file path. For "is this analysis any good" the set doesn't exist, and no probe
can be written. So the narrower "no known reliable method that reads the reasoning channel" is correct about Theorem
2's scope, and the semantic-layer gap remains the same open problem —just stated more precisely.
This is the clean version, and the enumerable-set requirement is doing more work than the text-versus-outcome-channel framing that sits on top of it. The probe isn't independent because it reads a different channel. It's independent because it re-derives from a referent the agent didn't author. When the affected set is enumerable, the outcome channel just is a second author, a namespace that can be recomputed without trusting the explanation. That's the whole mechanism.
Which means the channel distinction is downstream, not fundamental. When the proposition has no author-independent referent, "is this analysis any good," there's nothing to re-derive from, so every verifier collapses back to reading the reasoning channel, because for that proposition the reasoning channel is the only channel that exists. Outcome-channel verification doesn't fail there because it's the wrong channel. It fails because there's no referent to point the second author at.
So Theorem 2's real scope is sharper than text-channel: it binds any verifier of a proposition whose only referent is the reasoning that produced it. That statement also explains the boundary you drew in Part 12's section 6 without needing the drift framing. The gap closes exactly where an enumerable set exists and stays open exactly where it doesn't, and the reason is the same fact stated once instead of twice.
You're right, and this is cleaner than what I wrote. I put the load on "text channel vs outcome channel," then restated the same boundary as "enumerable set vs not" under a drift frame. One fact, said twice. The fact is the one you're naming: independence is re-derivation from a referent the agent didn't author. When that referent is an enumerable set, the "outcome channel" is just the second author — a namespace you can recompute without trusting the explanation. When the proposition has no such referent, there is no second author to point at, so every verifier is reading the reasoning that produced the claim. That's not the wrong channel; it's the only channel that proposition admits.
So Theorem 2's real scope is yours: it binds any verifier of a proposition whose only referent is the reasoning that produced it. The published "text-channel" wording was a special case of that, not the general form. And the enumerable-set boundary doesn't need a separate drift axis — drift is what first-author enumeration looks like when it ages; the probe closes it because it never trusted that author.
I'll take that as the clean statement. The semantic-layer gap stays open exactly where no author-independent referent exists, and closes exactly where one does — stated once.
The part I like here is separating “did the output satisfy the contract?” from “does the model think it is done?” Those are very different questions.
One extra guard I’d want is a cheap stuck-loop budget: same failing red line N times means stop and surface the evidence, not keep sampling. Otherwise a deterministic gate can still become an expensive infinite patience machine.
You're right, and the article flagged exactly this gap — the "boundary of loops" section marked it as an untested hypothesis rather than a solved one. Your budget is the finer signal that distinguishes "still making progress, just not there yet" from "spinning on a conceptual error no amount of sampling will fix." The first deserves more steps; the second deserves to stop and hand the evidence to a human. Rule 3's hard cutoff stops on elapsed effort; same-failing-red-line-N-times stops on evidence that effort has stopped paying.
So I ran it. Two model tiers (deepseek-v4-flash, glm-5.2), two task classes under a red line — 3 repairable tasks, and 4 conceptual tasks where the test expectation contradicts the requirement's literal meaning, so iteration can't fix it. 8-step cap, signature-repetition budget at N=3.
The headline result: the budget works, but only where the failure signature is stable — and signature stability is model-dependent.
On glm-5.2, the one conceptual task that genuinely stuck produced a single, stable failure signature every step (it kept emitting the same wrong answer verbatim). The budget fired at step 3 and saved 5 steps of pointless sampling on that task — exactly your "stop and surface the evidence." (The other three conceptual tasks on glm-5.2 converged in 2–4 steps — the model stumbled onto the test's hidden intent for those — which is why the per-model average saved is only 1.5.) On deepseek-v4-flash, all four conceptual tasks oscillated between two failure signatures (a wrong answer, then a
NameErrorfrom rewriting, then the wrong answer again) — so no single signature repeated three times consecutively, and the budget never fired. It degraded gracefully back to the step-cap, which is the honest fallback.The repairable side was the cleanest result: 0% false-stops on both models. When a task is genuinely fixable, the model converges in 1–2 steps and the budget never triggers — so it doesn't kill work that would have succeeded.
The conclusion I'd draw is narrower than "your budget works": the budget works when the model's stuck behavior is stereotyped, and silently no-ops when the model oscillates. That's worth knowing because it tells you when the cheap mechanism pays for itself (stable-stuck models) and when you're paying for it without benefit (oscillating models, where you still need the step-cap as backstop). The oscillation case is the real open question — a signature that captures "same failure class" rather than "same literal signature" might close it, but that's the calibration knob, and I haven't tuned it.
The one thing I'd push back on slightly: "cheap" describes the mechanism, not the calibration. Picking N and the similarity threshold is a scalar-tuning problem with the same distribution-shift risk as any feedback loop. But the failure domain is narrow (a scalar, bounded), which makes it a good trade.
Experiment script + results: stuck-loop-budget-test.py — I also updated the article's boundary-of-loops section with this data.