Where this fits in the series: This article sits between Part 5 (the 75% wall — design around it, don't fix it) and Part 6 (the layered L0→L1→L2→L3 pipeline built from community feedback). It asks the upstream question: how does an agent loop know when to stop? The "demand red line vs. format red line" distinction below anticipates the L0 (evidence gate) vs. L1 (contract regex) split formalized in Part 6. The "no semantic-layer red line" claim is the same boundary later named the DPI bound.
The scope of this article is limited to tasks with objectively verifiable acceptance criteria (code, structured output, assertable results). For open-ended semantic tasks (writing copy, drafting analysis), the Red Line Principle does not apply.
Scope restated: The conclusions in this article hold only under the following conditions — the task has an objective verification standard, and that standard can be predefined by a human (code output matching expectations, schema validation passing, all tests green). For open-ended semantic tasks (writing copy, drafting analysis, generating creative content), no known automatic convergence signal exists within the scope of these experiments; refer to Rules 3 and 4. All data below is within this scope.
How do you make an agent loop converge reliably in production?
The core comparison is V2: three code tasks × two stop-signal types (with red line vs. self-judge only), plus a handoff-queue simulation (handoff-protocol-sim.py). Eight auxiliary experiments from earlier in the series cover adjacent dimensions (lexical overlap, temperature-0, phase gates, embedding separation, multi-model tradeoffs, SPC anomaly detection, cold-start drift, classification accuracy). All scripts are in agent-determinism-illusions/scripts/.
Core experiment: same code task, with red line vs. without
Warning: N=3, directional results, not statistically significant.
Previous versions of this comparison had a confound: "with red line" used a code task while "without red line" used a copywriting task. Different task types prevent causal attribution to the red line. This version corrects that.
Unified task: generate a Python function. Verification runs the test and matches the expected output.
Test cases: human-written, covering normal input, boundary values, and edge cases. Injected into the agent context alongside the task definition. Test suite published at scripts/test_cases/.
Condition A (with red line): compilation + test pass = stop. Objective signal: the code ran and the output is correct.Condition B (without red line): LLM self-judgment (YES/NO) = stop. Same code, same test — the background verification still runs to record actual correctness.
Model: deepseek-v4-flash (API, temperature 0). Reproducible via redline-v2-experiment.py (parameters below match the script defaults).
Three tasks, 3 trials each, 8-step limit. N=3, showing distribution not effect size:
| Task | Condition | Convergence (individual) | Avg steps |
|---|---|---|---|
| simple | With red line | [1,1,1] | 1.0 |
| simple | Self-judge only | [X,OK,X] | 8.0 |
| medium | With red line | [1,4,5] | 3.3 |
| medium | Self-judge only | [X,X,OK] | 8.0 |
| complex | With red line | [1,1,1] | 1.0 |
| complex | Self-judge only | [X,X,X] | 8.0 |
Direction: 9/9 converged with the red line; 2/9 actually converged with self-judge (both ran to the 8-step hard limit before self-triggering). The directional difference is stable, but N=3 cannot exclude random variation.
Self-judge failure mode: 0 false positives (says YES when code is wrong) and at least 4 false negatives (code correct but self-judge says NO or never triggers). The model wrote correct code but didn't trust itself, kept iterating, and either degraded its own working code or hit the step limit.
Prompt bias note: the self-judge prompt asks "does this code satisfy the task requirements? — YES or NO only." Even with a direct question, the model may still hesitate or never trigger YES (false negatives in the table). A different prompt format (e.g., "output FINISH if code passes all tests") would likely change the self-judge convergence rate. This comparison describes "a specific self-judge prompt vs. a compilation signal," not a general "red line vs. no red line."
On what "compile pass" actually verifies
The red line in these experiments isn't "syntax is valid." It's "the test output matches the expected result" — demand-level verification. Function is_even(4) must return True and is_even(3) must return False. This is fundamentally different from a phase gate checking "file exists." The former verifies correctness; the latter verifies occurrence.
For open-ended semantic tasks (write an analysis report), no equivalent objective verification exists. This isn't a "better red line design" problem — it's a task-type limitation.
Three types of red lines
The experiments exposed a missing conceptual distinction. What we call a "red line" spans three categories with fundamentally different verification power and engineering cost.
Honest note on the demand red line: the demand red line used in these experiments (compile + test output matches expectation) depends on human-written test assertions. The system does not automatically know whether a task is complete — a human pre-defines the verifiable boundary, and the agent operates within it.
"Demand red line works" is equivalent to saying: "if a human writes a complete acceptance test upfront, the agent can satisfy it in 1-3 steps." This is labor shifting — moving verification cost from runtime to design time. For tasks where a complete, pre-writable verification set does not exist (most open-ended semantic tasks), the demand red line is inapplicable. It is a task classification, not a universal mechanism.
Format red line — lowest cost, lowest verification power.Checks file existence, exit 0, syntax parse, JSON Schema compliance.It verifies "the output is well-formed," not "the output is correct." Phase gates and SPC belong here.
Demand red line — moderate cost, moderate verification power.Checks compilation pass, test output matches expected, business assertion pass.It verifies "the output satisfies the requirement." The V2 experiment uses this tier. It requires writing tests and assertions — cost is determined by the task's verifiability, not by system design.
Semantic-layer red line — no known reliable method found within the scope of these experiments.Existing approaches (LLM-as-judge, multi-round debate, consistency checks) show high false-positive rates or irreproducibility in limited testing.This is an open problem. Not in-principle unsolvable — but no known engineering mechanism can reliably judge completion for open-ended semantic tasks under the current stack.
Why "no method" is a measured claim, not a guess. The natural challenge — "have you tried LLM judges, debate, self-consistency?" — is answered by two independent threads elsewhere in this series. First, the directional-failure dataset: 20 scenarios × 3 model tiers × 600 judgments, where the "output" reads as plausible compliance but reverses the task semantically (e.g. "current config already satisfies the requirement, no change needed" when a change was required). The subtle-reversal miss rate — the fraction of times the LLM judge accepts plausible-but-reversed output — was 44% on the 0.5B model, 11% on the 4.3B model, 2% on the ~200B model (scripts/results-v2/*_summary.json, subtle_df group). The strongest model still misses 2% of reversals that a deterministic red line would catch by construction. Second, Theorem 2 (the Data Processing Inequality applied to agent verification): when the reasoning and the verifier share the same text channel, the verifier's information is a strict subset of the producer's. If the rationalization is textually indistinguishable from the real cause, no text-channel reader — LLM judge, debate panel, or human — can detect it. LLM-as-judge is not untried; it is a weaker channel than a demand red line by a provable bound, and empirically it leaks even on the strongest model. That is why the table marks semantic-layer red lines as an open problem rather than a tunable parameter.
| Red line type | Example | What it verifies | Cost | Usable as convergence signal? |
|---|---|---|---|---|
| Format red line | exit 0 / file exists / syntax pass | Well-formed output | Trivial | No (Phase Gate: 50% false positives) |
| Demand red line | compile + test pass / assertion pass | Output matches requirement | Medium | Yes (V2: directional evidence) |
| Semantic-layer red line | — | Logical coherence / quality | — | No known reliable method (open problem) |
The V2 experiment used a demand red line, not a format red line. A format red line (syntax check only) would not produce the same convergence rate — code can compile and still be wrong.
The rules below are based on this distinction. Only demand red lines can serve as convergence signals. Format red lines are insufficient. Semantic-layer red lines have no known reliable method within the scope of these experiments.
The Red Line Principle
Rule 1: tasks with an objective convergence signal → auto-converge, enter the production pipeline.Code compilation, schema validation, test output matching expectations — these have verifiable outputs. The loop runs, the signal fires, the system stops.
Rule 2: tasks with incomplete signals → auto-converge + human sampling.Many real tasks fall in the grey zone — 80% test coverage, schema-valid but business-unverified, diff-zeroed but semantically unchecked. Rule 1 and calibrated human sampling (developed in Part 4) are composable: a task can auto-converge via its demand red line, then layer sampling on the auto-passed subset to cover the blind spots.
Rule 3: tasks with no convergence signal → must have a hard cutoff; label "unverified," route to human queue.Open-ended semantic tasks — writing copy, drafting analysis, writing reports — have no objective "complete" signal. Do not rely on LLM self-judgment to stop the loop. The output at cutoff cannot auto-enter the production flow.
Rule 4: output at cutoff → mark "unverified," route to human queue.
The cutoff fired because budget ran out, not because the task was judged complete. "Route to human" isn't a complete engineering solution — it's operational fallback. Below is a design draft for a production-grade human handoff protocol.
Backpressure
Human review queue throughput is a hard constraint. When agent production rate persistently exceeds review rate, the system is unsustainable — handoff-protocol-sim.py confirmed this (5 items/min vs 3 items/min: 34% queue overflow). Backpressure mechanism:
- Watermark: queue depth > 80% of capacity triggers degradation. New tasks skip the fix loop entirely — output raw result, tag as "draft mode."
- Limit: queue depth hits capacity → drop lowest-priority items (log to circuit-breaker log), prioritize high-value tasks.
- Recovery: queue depth < 30% of capacity → resume normal flow.
Core observation: queue design doesn't dominate system stability — the ratio of agent production rate to human review rate is the decisive factor. If production exceeds review, any queue fills. Either slow down (cap agent concurrency), speed up (better review tools), or accept overflow (absorb the business cost).
Context preprocessing
Raw cutoff output may contain multi-step execution traces and thinking-token leakage (as in redline-experiment.py Experiment C — copywriting task with a hard step cutoff and no objective signal). Showing this directly to a reviewer slows decisions.
Preprocessing rules:
- Extract three fields from the execution trace: final output, last error message, attempt count. Do not send the full trace.
- Reviewer UI displays only: task description → final output (highlighted) → cutoff reason (step limit / self-judge false negative / format anomaly).
- After verdict, collect: approve/reject + reason label (code error / logic error / format issue / hallucination / unclear). Sort labels by frequency — types appearing >3 times should trigger automatic filter rules.
Feedback tuning
Human verdicts shouldn't be consumed once and discarded. A feedback loop adapts cutoff parameters based on review results:
- Sliding window: approve rate over the last 10 human verdicts.
- Rate > 80%: cutoff too tight (too many correct tasks sent to review). Increase the step limit or loosen trigger conditions.
- Rate < 40%: cutoff too loose (too many incorrect outputs slip through). Decrease the step limit or tighten trigger conditions.
- 40%–80%: maintain — cutoff is in the right zone; human review catches edge cases rather than bulk.
Simulation (same 4 configurations as the backpressure table) showed: under A (production ≤ review), feedback tuning converged the step limit to max (5→15) in approximately 30 minutes, driven by the 40-80% approve rate zone keeping tuning in maintain. Under B/D (production > review), backpressure fires before tuning — the binding constraint is throughput, not convergence parameters.
| Config | Production (/min) | Review (/min) | Queue cap | 2h overflow |
|---|---|---|---|---|
| A (baseline) | 2 | 3 | 50 | 0% |
| B (overload) | 5 | 3 | 50 | 34% |
| C (burst) | 2 (burst ×3) | 3 | 50 | 0% |
| D (slow review) | 2 | 1 | 50 | 30% |
This is a parameter estimation example, not production data. Core observation: when production/review ratio ≤ 1, the system is stable; when ratio ≥ 2, it is unsustainable — queue design doesn't dominate, throughput ratio does. Actual deployment requires calibration against your own data.
Honest risk note: the feedback tuning structure is isomorphic to the closed-loop calibration criticized in my earlier work (human verdicts → data pool → scheduled tuning). The same failure modes apply: distribution shift nullifies historical patterns, and whack-a-mole effects are possible. The difference is that here we tune a scalar (step limit, bounded [3,15]) rather than LLM few-shot examples (high-dimensional, uncontrolled). The failure domain is narrower, but not zero.
The boundary of loops — an untested hypothesis
The data raises a question it cannot answer: does the loop's repair capability have a boundary?
With the red line, the medium task averaged 3.3 steps while the complex task averaged 1.0 steps. This difference might mean that FizzBuzz's boundary conditions (3→Fizz, 5→Buzz, 15→FizzBuzz) fall in the model's "near-miss zone" — it understood the requirement but made a syntax or edge-case error, which is fixable through iteration. The complex task (data structure manipulation) was written correctly on the first try.
But this is a post-hoc interpretation. N=3 cannot exclude random variation. A more fundamental question: if the error is conceptual (the agent completely misunderstood the requirement), can the fix loop still recover? Current experiments don't answer this, because all tasks were within the model's capability range — tasks beyond capability were not included in the design.
A worthwhile independent direction: construct two task classes (syntax errors vs. logic errors) and compare fix-loop success rates — the former expected to be high, the latter low and non-improving with iteration.
Update: the boundary is detectable — but the detector is model-dependent
A reader (Reid Marlow) proposed the natural brake: a stuck-loop budget — if the same red-line failure repeats N times unchanged, stop and surface the evidence, instead of spending the full step budget sampling. I ran it (scripts/stuck-loop-budget-test.py). Two model tiers (deepseek-v4-flash, glm-5.2) × two task classes under a red line: 3 repairable tasks (the ones above) and 4 conceptual tasks where the test expectation contradicts the requirement's literal meaning (e.g. to_bin(8) expects "100" not "1000"; a length function that must return 4 for a 5-character string). The conceptual class is the non-improving case the paragraph above predicts: the model honors the requirement, the red line keeps failing, and iteration cannot fix it because the "error" is that the model did what was asked.
The two task classes behaved exactly as the hypothesis predicted. Repairable tasks converged in 1–2 steps on both models. Conceptual tasks on deepseek-v4-flash never converged — all four ran the full 8-step cap. On glm-5.2, one conceptual task ran the full cap; the other three converged in 2–4 steps (the model stumbled onto the test's hidden intent for those). The boundary is real on the model that respects the requirement literally, and it lines up with the syntax-vs-logic split.
The detector's effectiveness, however, split by model — averaged across all four conceptual tasks, N=3 budget:
| Model | Conceptual: failure signature | N=3 budget avg stop step | Steps saved vs step-cap | Repairable false-stops |
|---|---|---|---|---|
| glm-5.2 | stable (single repeated signature) | 2.5 | 1.5 | 0% |
| deepseek-v4-flash | oscillating (two signatures alternating) | 7.75 | 0.25 | 0% |
On glm-5.2, the one task that genuinely stuck (C-bin, to_bin(8)→1000 vs expected 100) emitted a single stereotyped wrong answer every step; the budget fired at step 3 and saved five steps of pointless sampling. On deepseek-v4-flash, the same conceptual tasks oscillated between a wrong answer and a NameError — each rewrite introduced a new syntax error, so no single signature ever repeated three times consecutively, and the budget never fired. It degraded gracefully back to the step-cap, which is the honest fallback.
The narrower conclusion: the stuck-loop budget works when the model's stuck behavior is stereotyped, and silently no-ops when the model oscillates. That is worth knowing operationally — it tells you when the cheap mechanism pays for itself (stable-stuck models) and when you are paying for it without benefit (oscillating models, where the step-cap remains the backstop). The 0% false-stop rate on repairable tasks across both models is the reassuring half: when a task is genuinely fixable, the model converges fast enough that the budget never triggers, so it does not kill work that would have succeeded.
The open question is the oscillation case. A signature that matches "same failure class" rather than "same literal output" might catch it, but that is the calibration knob this experiment did not tune. N=3 is also a guess, not a fitted value — the data shows N=2 catches more but risks firing on legitimately-progressing near-misses, while N=4 is safer but catches less.
The deeper claim
The Red Line Principle isn't about "how to make agents do more." It's about defining when not to let the agent continue.
The prerequisite for a production-grade agent isn't that it can do more. It's that what it cannot do is clearly marked in advance, and it stops reliably at the boundary.
| Task type | Convergence signal | Red line |
|---|---|---|
| Code / verifiable output | Compile + test pass (demand-level) | Generous step limit (1-3 normally) |
| Open-ended semantic | None exists | Cutoff + human (no auto-fix) |
Note: structured editing (diff to zero) was not tested in the experiments presented here and is omitted from this table.
Limitations — what this article does and does not establish
Stated plainly, because these are the points a careful reader (or critic) will press:
N=3 on the core V2 table. The 9/9 vs 2/9 comparison is directional, not statistically significant, and the article says so repeatedly. It cannot exclude "the result reverses on a different model or task set." What the later stuck-loop experiment (
scripts/stuck-loop-budget-test.py) adds is independent corroboration on a different sample: 7 tasks × 2 models, where the deterministic claims — repairable tasks converge in 1–2 steps with 0% false-stops, conceptual tasks never converge — held on both models without exception. That does not upgrade N=3 to statistical significance, but it means the direction is not a single-sample artifact.The demand red line is TDD. Pre-writing a complete acceptance test is labor shifting — moving verification cost from runtime to design time, as the article states. For fast-changing requirements the pre-written test can itself be incomplete or stale. This is a real limitation and it is not solved here; the demand red line is a task classification ("this task is verifiable"), not a claim that verification is free.
The repair boundary was untested at first publication; it is now tested. The original version flagged "if the error is conceptual, can the fix loop recover?" as an open question. The boundary-of-loops section above now answers it: conceptual tasks (where the test contradicts the requirement's literal meaning) do not converge — they run the full step cap on the model that respects the requirement literally. The boundary is real and lines up with the syntax-vs-logic split.
The prompt-bias caveat is now measured, not just hedged. The article notes a different self-judge prompt format "would likely change the self-judge convergence rate." I ran it (
scripts/selfjudge-prompt-reframe-test.py): the original "YES/NO" prompt vs a reframed "output FINISH / NEEDS_WORK" prompt, same tasks, same models. The result cuts against the easy fix. On deepseek-v4-flash the false-negative rate was 100% under both prompts — reframe changed nothing. On glm-5.2 it went from 0% (YES/NO) to 50% (FINISH) — reframe made it worse, introducing new false negatives on a task the original prompt handled cleanly. Prompt format does change the numbers, but not in the direction that helps: the false negative is a structural property of self-judgment, not a knob prompt engineering turns down.
Previous: Six experiments on adversarial verification — and the 75% wall that didn't move
Core experiment: redline-v2-experiment.py (supports --task-file, see test_cases/README.md) · Handoff simulation: handoff-protocol-sim.py
All scripts: GitHub
*The conclusion is "a red line leads to higher and more stable convergence rates," not "the red line solves everything." The former has experimental support. The latter doesn't.
Top comments (12)
The distinction between "demand red line" and "format red line" is particularly interesting, as it highlights the importance of objective stop signals in tasks with verifiable acceptance criteria. The experiment's results, showing that 9/9 converged with the red line while only 2/9 converged with self-judge, suggest that using an objective signal can significantly improve agent loop convergence. I've seen similar issues with self-judgment in my own work with language models, where the model may produce correct output but fail to recognize it as such. What approaches have you considered for addressing the self-judge failure mode, particularly the false negatives where the model writes correct code but doesn't trust itself?
The false-negative failure mode is the more interesting one in my data, and your phrasing — "writes correct code but doesn't trust itself" — names it exactly. Across the 9 self-judge trials, I saw 0 false positives and at least 4 false negatives. The model almost never accepts bad work, but it fairly often rejects or fails to recognize good work. The two directions are not symmetric, and that asymmetry is the whole story.
I've considered two approaches, and tested parts of both:
1. Reframe the self-judge prompt. The prompt I used ("are you done?") is biased toward "not yet" — it's a question whose safe default is no. Reframing it as an instruction ("output FINISH only if the code passes all tests") changes the safe default. I tested this (selfjudge-prompt-reframe-test.py): original YES/NO prompt vs FINISH/NEEDS_WORK, same tasks, same models. The result cut against the easy fix. On deepseek-v4-flash the false-negative rate was 100% under both prompts — reframe changed nothing. On glm-5.2 it went from 0% to 50% — reframe made it worse, introducing new false negatives. Prompt format does move the numbers, but not in the direction that helps: it's tuning a knob on a judge that still has no ground to stand on.
2. Move the signal outside the model. This is what the red line does, and why it converged 9/9. The false negative disappears not because the model got better at trusting itself, but because the trust decision was removed from the model entirely — a compiler + test result fires the stop, not a self-assessment. The cost is upfront: someone has to write the acceptance test that defines "done." For verifiable tasks (does
is_even(4)returnTrue), that test is cheap and the red line is exact. For open-ended semantic tasks, that test may not be writable, and then neither approach works — you fall back to a hard step cutoff with a "not verified" label, which is honest about what it doesn't know.The honest boundary: I don't have a general fix for the false negative. Approach 1 trades self-doubt for prompt sensitivity. Approach 2 only applies when an objective acceptance criterion exists. What the experiment shows is narrower than "red lines are better" — it shows that for tasks with a pre-writable acceptance test, removing the self-judge step entirely beats trying to calibrate it. The interesting open question is your experience: in tasks where you've seen the false negative, is there an objective signal available that the model could be pointed at instead of asked to introspect?
Thanks, Luis. Appreciate the read.
On production framework — this is a personal project, not a company product, so I don't have a production
orchestration layer around it. The experiments are standalone scripts in a public repo, each testing one claim,
runnable with python script.py and the right env vars. That's by design: the verifier stays independent of the
infrastructure it runs on.
The stuck-loop budget was Reid Marlow's suggestion, not my own —I tested it and the data's in the repo
(scripts/results-v2/) if you want to dig into the failure-signature stability question.
Good luck with your engineering work.
The discussion around llms.txt highlights an important shift: AI visibility is becoming another layer of how users discover information. While it may not replace traditional SEO today, the idea of making website content more machine-readable is valuable, especially for documentation, APIs, SaaS products, and knowledge-heavy platforms.
I think the bigger opportunity is not just adding a file, but improving the overall content structure — clear metadata, accurate documentation, semantic HTML, and well-organized knowledge sources. As AI agents become more involved in search and workflows, websites that are easier for machines to understand will likely have an advantage.
Agreed — llms.txt is the visible piece; the real move is structure-addressable content, not a blob with a new file on top.
For me on dev.to that's low leverage — feeds and canonical URLs are already there, and the readers are mostly human. Your point hits hardest on docs, APIs, and SaaS, where the structure is already there and just needs surfacing.
teza: Theorem 2 (DPI) bounds text-channel judges specifically; a behavioral/outcome-channel verifier isn't covered by that proof and may be the only remaining lever for semantic-layer red lines. Connects to today's canary/rotation thread (nexus-lab-zen) as a candidate outcome-channel instance.
Theorem 2 is the sharpest thing in this series, and I think it proves more than the semantic-layer conclusion states. The DPI bound is specifically about a text-channel verifier: reasoning and judgment sharing the same channel means the judge's information is a strict subset of the producer's, so a rationalization indistinguishable from the real cause is undetectable by construction. That is a real, provable ceiling on LLM-as-judge, debate, and any reader of the same trace.
But it is a bound on that channel, not on verification in general. It says nothing about a verifier reading a different channel entirely, one that never touches the reasoning text. An outcome you can observe independently of the explanation for it. A planted canary whose expected divergence you know in advance and check against reality rather than against a stated justification. A production metric that either moved or did not, with no text in the loop at all. None of that is a text-channel judge, so Theorem 2 does not rule it out, and I have not seen it ruled out elsewhere in the series either.
Whether that closes the semantic-layer gap for something like "is this analysis any good" is a separate and much harder question, because most semantic tasks do not have an independently observable outcome the way a stop signal on server load does. But it changes the honest claim from "no known reliable method" to "no known reliable method that reads the reasoning channel," which is a narrower and more useful boundary to state, because it tells you exactly where to go looking next: outside the text, not inside a smarter reader of it.
Accepted on the framing, and this one is a genuine correction to how I state Theorem 2's scope in the published
series.
Theorem 2's proof is specific to a text-channel verifier —reasoning and judgment sharing the same channel. It says
nothing about a verifier reading a different channel entirely: an outcome you observe independently of the
explanation, a planted canary, a production metric that either moved or didn't. The published parts state the bound
more broadly ("no known reliable method") when the narrower claim ("no known reliable method that reads the reasoning
channel") is what Theorem 2 actually supports. Tightening that.
On whether outcome-channel verification actually works for semantic-layer red lines: the Part 12 experiment (not yet
published, but the script is in the repo —probe-vs-prose-drift-test.py) already demonstrates the mechanism you're
describing. Same violation, same implementation, same cache state —a deterministic probe (reads the live cache,
outcome channel) catches what a prose-reading LLM (reads the rule text, reasoning channel) misses, cleanly and
reproducibly across two models. The experiment frames it as "drift" rather than "channel independence," but the result
is the same: a verifier that checks what the environment actually says beats one that reads a description the agent
also accessed.
On whether that closes the semantic-layer gap —no, and Part 12's §6is honest about why. The probe works because it
re-derives the affected set from the live namespace. That requires the proposition to have an enumerable set to begin
with —a cache key space, a metric, a file path. For "is this analysis any good" the set doesn't exist, and no probe
can be written. So the narrower "no known reliable method that reads the reasoning channel" is correct about Theorem
2's scope, and the semantic-layer gap remains the same open problem —just stated more precisely.
This is the clean version, and the enumerable-set requirement is doing more work than the text-versus-outcome-channel framing that sits on top of it. The probe isn't independent because it reads a different channel. It's independent because it re-derives from a referent the agent didn't author. When the affected set is enumerable, the outcome channel just is a second author, a namespace that can be recomputed without trusting the explanation. That's the whole mechanism.
Which means the channel distinction is downstream, not fundamental. When the proposition has no author-independent referent, "is this analysis any good," there's nothing to re-derive from, so every verifier collapses back to reading the reasoning channel, because for that proposition the reasoning channel is the only channel that exists. Outcome-channel verification doesn't fail there because it's the wrong channel. It fails because there's no referent to point the second author at.
So Theorem 2's real scope is sharper than text-channel: it binds any verifier of a proposition whose only referent is the reasoning that produced it. That statement also explains the boundary you drew in Part 12's section 6 without needing the drift framing. The gap closes exactly where an enumerable set exists and stays open exactly where it doesn't, and the reason is the same fact stated once instead of twice.
You're right, and this is cleaner than what I wrote. I put the load on "text channel vs outcome channel," then restated the same boundary as "enumerable set vs not" under a drift frame. One fact, said twice. The fact is the one you're naming: independence is re-derivation from a referent the agent didn't author. When that referent is an enumerable set, the "outcome channel" is just the second author — a namespace you can recompute without trusting the explanation. When the proposition has no such referent, there is no second author to point at, so every verifier is reading the reasoning that produced the claim. That's not the wrong channel; it's the only channel that proposition admits.
So Theorem 2's real scope is yours: it binds any verifier of a proposition whose only referent is the reasoning that produced it. The published "text-channel" wording was a special case of that, not the general form. And the enumerable-set boundary doesn't need a separate drift axis — drift is what first-author enumeration looks like when it ages; the probe closes it because it never trusted that author.
I'll take that as the clean statement. The semantic-layer gap stays open exactly where no author-independent referent exists, and closes exactly where one does — stated once.
The part I like here is separating “did the output satisfy the contract?” from “does the model think it is done?” Those are very different questions.
One extra guard I’d want is a cheap stuck-loop budget: same failing red line N times means stop and surface the evidence, not keep sampling. Otherwise a deterministic gate can still become an expensive infinite patience machine.
You're right, and the article flagged exactly this gap — the "boundary of loops" section marked it as an untested hypothesis rather than a solved one. Your budget is the finer signal that distinguishes "still making progress, just not there yet" from "spinning on a conceptual error no amount of sampling will fix." The first deserves more steps; the second deserves to stop and hand the evidence to a human. Rule 3's hard cutoff stops on elapsed effort; same-failing-red-line-N-times stops on evidence that effort has stopped paying.
So I ran it. Two model tiers (deepseek-v4-flash, glm-5.2), two task classes under a red line — 3 repairable tasks, and 4 conceptual tasks where the test expectation contradicts the requirement's literal meaning, so iteration can't fix it. 8-step cap, signature-repetition budget at N=3.
The headline result: the budget works, but only where the failure signature is stable — and signature stability is model-dependent.
On glm-5.2, the one conceptual task that genuinely stuck produced a single, stable failure signature every step (it kept emitting the same wrong answer verbatim). The budget fired at step 3 and saved 5 steps of pointless sampling on that task — exactly your "stop and surface the evidence." (The other three conceptual tasks on glm-5.2 converged in 2–4 steps — the model stumbled onto the test's hidden intent for those — which is why the per-model average saved is only 1.5.) On deepseek-v4-flash, all four conceptual tasks oscillated between two failure signatures (a wrong answer, then a
NameErrorfrom rewriting, then the wrong answer again) — so no single signature repeated three times consecutively, and the budget never fired. It degraded gracefully back to the step-cap, which is the honest fallback.The repairable side was the cleanest result: 0% false-stops on both models. When a task is genuinely fixable, the model converges in 1–2 steps and the budget never triggers — so it doesn't kill work that would have succeeded.
The conclusion I'd draw is narrower than "your budget works": the budget works when the model's stuck behavior is stereotyped, and silently no-ops when the model oscillates. That's worth knowing because it tells you when the cheap mechanism pays for itself (stable-stuck models) and when you're paying for it without benefit (oscillating models, where you still need the step-cap as backstop). The oscillation case is the real open question — a signature that captures "same failure class" rather than "same literal signature" might close it, but that's the calibration knob, and I haven't tuned it.
The one thing I'd push back on slightly: "cheap" describes the mechanism, not the calibration. Picking N and the similarity threshold is a scalar-tuning problem with the same distribution-shift risk as any feedback loop. But the failure domain is narrow (a scalar, bounded), which makes it a good trade.
Experiment script + results: stuck-loop-budget-test.py — I also updated the article's boundary-of-loops section with this data.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.