DEV Community: Q00

A verifier should add evidence, not rewrite history

Q00 — Tue, 18 Aug 2026 19:34:39 +0000

A verifier should add evidence, not rewrite history

I spent the last few weeks closing a loop around an AI coding agent.

The first version had an obvious flaw: the worker could see the command and expected output that would grade its work. A stuck agent could optimize for the assertion string instead of the requirement. PR #1916 removed that answer key from the worker-facing success contract and connected one invocation through run, evaluation, and bounded evolution. RFC #1917 records the design.

That change removed the answer key, but it exposed a separate design question: how much authority should the verifier have over facts that already happened?

A worker's completion status and a later machine check are separate facts. The worker either completed the acceptance criterion or it did not. The machine check may then produce supporting evidence, find a real failure, or fail to run. Collapse those facts into one boolean and the loop can report an outcome that never occurred.

Ouroboros v0.51.12 uses one rule at that boundary:

Machine verification is monotonic: it adds evidence to a successful worker result, but it cannot recover a failed worker result or turn completed work into a retry when the verifier is unavailable.

The implementation is PR #2187. The release is v0.51.12.

The four outcomes

The rule has four outcomes:

Worker result	Machine verification	Final meaning
failed	anything	worker failure stays failed
succeeded	passed	verified success
succeeded	failed	rejected by machine evidence
succeeded	unavailable	unverified success, no retry

A passing command cannot rescue failed work

Earlier versions treated the presence of a verify_command as a reason to relax some transcript evidence. That created a perverse result: adding exit 0 could make an acceptance criterion weaker than one with no command.

The current evidence classifier discards verifier-specific exemptions. Its docstring states the invariant directly:

A verify command is additive evidence. Declaring one never removes the
worker transcript obligations; this keeps a vacuous command from making
an acceptance criterion weaker than a contract-less one.

When the active execution profile requires files_touched, commands_run, or tests_passed, each still needs its own support. The verify command runs afterward and adds one more observation.

The execution path enforces that order. _run_ac_verify_gate is called only when the worker result is already successful:

if success and verify_gate_active and has_success_contract:
    verify_gate_outcome = await _run_ac_verify_gate(...)

A green command cannot convert failed work into success because the command does not run in that branch.

Missing verifier infrastructure cannot manufacture new work

Suppose the worker completed the task, but the machine has no Bash, the verifier process cannot start, or it times out before producing a judgment.

A worker retry fixes none of those conditions. It only repeats completed work to compensate for infrastructure the worker does not own.

v0.51.12 records that case as UNAVAILABLE. The typed contract is strict:

UNAVAILABLE requires TRANSCRIPT_MISSING_INFRASTRUCTURE and ACCEPT

ACCEPT permits execution to continue without retrying the worker; it does not claim that verification passed. The final report carries the result as unverified success.

Recording an unavailable verifier as worker failure would send the loop back into code generation, where it could overwrite correct work while leaving the missing verifier untouched.

A real failed check still rejects the result

A resolved verifier that runs and returns a nonzero status rejects the successful worker result:

if returncode != 0:
    return VerifyGateOutcome(
        passed=False,
        reason=f"verify_command exited with status {returncode}",
    )

An output_assertion missing from the combined output also rejects the result. Real negative evidence can move a successful result to rejected. It cannot move a failed worker result upward or retry the worker because the verifier could not judge.

The state transitions are:

worker failure ------------------------------> failure
worker success + verifier failure ----------> rejection
worker success + verifier pass -------------> verified success
worker success + verifier unavailable ------> unverified success

The shell is part of the judgment boundary

A verification command is only meaningful if the runtime executes the command that the acceptance contract declared.

The new path resolves an absolute Bash implementation and executes the original text through bash -c. It does not translate the command for cmd.exe, substitute sh, invoke the WSL launcher against another filesystem, or fall back to a shell emulator.

The command text stays unchanged. If a compatible Bash does not exist, the outcome is UNAVAILABLE rather than an approximation presented as evidence.

The process boundary also strips environment controls that can alter a verdict without changing the command or workspace. The list includes:

BASH_ENV, exported Bash functions, shell option state, and compatibility controls
PYTHONPATH, PYTHONSTARTUP, and PYTHONHOME
PYTEST_ADDOPTS and PYTEST_PLUGINS
NODE_OPTIONS
dynamic-loader preload hooks

PYTEST_ADDOPTS="--collect-only" is a small example. A test command can exit successfully without running the tests the contract author thought it ran. An exported function named pytest can replace the executable entirely. Both are outside the command text, so the verifier removes them before judgment.

Timeout and cancellation handling contain the verifier process tree. On POSIX, the process starts in a new session. On Windows, it is assigned to a kill-on-close Job Object before it is resumed. A timed-out check should not leave a background child mutating the workspace after the verdict has been recorded.

What this still does not prove

A real Bash process and a sanitized environment do not create a sandbox.

The workspace can still contain conftest.py, pytest.ini, a project-owned virtual environment, or a test that passed before the requested change existed. A command such as test -f out.txt proves that a path exists, not that the artifact satisfies the acceptance criterion.

The RFC keeps negative-control and differential-probe ideas as future evaluation research. They stay outside the ooo run critical path because arbitrary probes need real isolation. Running destructive baseline commands against the live workspace would give a verifier more authority while making its evidence less trustworthy.

v0.51.12 therefore limits the critical path to claims the current runtime can defend:

preserve the worker's execution facts
require transcript evidence independently
run the declared command through a sealed real Bash
reject on real negative evidence
report UNAVAILABLE when no judgment occurred

Why this closes the earlier loop work

1916 and #1917 established three properties: the worker does not receive its criterion's answer key in the success contract, the orchestrator owns the judgment, and rejected results can feed a bounded run-to-evaluate-to-evolve chain.

Together, those properties make a convergence loop possible. #2187 limits what the judge may rewrite once that loop is running.

Exposing the answer key lets the agent optimize for it. Letting the verifier rewrite outcomes distorts the execution history. In either case, the score can improve while the evidence gets worse.

For agent loops that execute real work, I would use this ownership rule:

generation owns attempts
verification owns additional evidence
infrastructure failure owns neither

If you maintain an agent loop, take one completed run and trace all four cases through your state machine. Verify that downtime does not retry the worker, a passing command cannot erase a failed execution, and adding a command does not remove existing evidence obligations.

Source: Q00/ouroboros, PR #2187, v0.51.12.

An ambiguity gate for AI coding agents: measure the request before writing code

Q00 — Tue, 18 Aug 2026 01:27:41 +0000

Every AI coding agent I have used will happily start writing code from a one-line request. "Build a task management CLI" leaves the data model, storage, priority rules, and ID scheme for the model to guess, and each guess is defensible and wrong. You find out three files into review, and the fix costs a rewrite instead of an answer.

An ambiguity gate is the opposite default: the system measures how vague your request still is, and refuses to freeze a spec until the number clears a threshold. This post describes the concrete gate we ship in Ouroboros (MIT, open source), with the actual constants, because when I went looking for prior art the search results were all academic papers and general advice. If other products implement this, I would genuinely like to compare notes.

The score

The interview loop asks questions and scores three axes of clarity, each in [0, 1]:

goal clarity, weight 0.40
constraint clarity, weight 0.30
success-criteria clarity, weight 0.30

Ambiguity is one minus the weighted sum. A request is gate-ready when ambiguity drops below 0.2. Each axis also has its own floor (0.75 goal, 0.65 constraints, 0.70 success criteria), so you cannot buy your way past a vague goal with very crisp constraints. The constants live in bigbang/ambiguity.py in the repo, not in a config file, on purpose: the gate is a product claim, and product claims should be greppable.

What the gate actually blocks, and what it does not

The gate is a default, not a wall. The UI shows a force option that generates the spec anyway. So the guarantee is narrower than "no vague spec exists". It is "no vague spec exists without someone explicitly choosing it". We think that is the right shape: the gate's job is to make vagueness a decision instead of an accident.

What changes in practice is where the iteration happens. Without a gate, iteration happens after code exists: you review, discover the wrong assumption, and re-prompt. With the gate, the same iteration happens in the interview, where a wrong assumption costs one answered question instead of one discarded implementation.

Why a number instead of judgment

An agent can always say "this seems clear enough". A threshold it must clear removes that discretion. To be precise about what the number controls: in the interactive interview there is no round limit and you decide when to stop asking; the 0.2 threshold gates the next step, generating the spec. In the unattended auto mode, a separate readiness constant (also 0.20, defined independently in the auto driver) decides when the interview is ready to hand off. Either way, a fixed number gates the step, not the model's own sense of "clear enough".

The number is not a quality guarantee. A request can score 0.1 and still describe a bad idea clearly. The gate only promises that you decided what goes into the spec.

Trying it

curl -fsSL https://raw.githubusercontent.com/Q00/ouroboros/main/scripts/install.sh | OUROBOROS_INSTALL_REF=devto-seo1 bash

Then, inside your coding agent, run these in order (ooo setup is a one-time step):

> ooo setup
> ooo interview "whatever you were about to prompt an agent with"

Try it with a request you were about to hand an agent. Reply with the starting score and the first question that exposed an assumption you had not decided yet.

Repo: github.com/Q00/ouroboros

당신의 에이전트 루프는 모델에게 커닝을 가르치고 있다

Q00 — Wed, 12 Aug 2026 04:50:05 +0000

AI 코딩 에이전트에 루프를 씌우는 건 단발 프롬프트가 막히면 누구나 떠올리는 다음 수입니다. 돌리고, 채점하고, 점수가 낮으면 다시 돌립니다.

그런데 바로 벽 두 개에 부딪힙니다.

점수는 오르는데 결과물은 여전히 틀립니다. 에이전트가 과제가 아니라 채점기를 만족시키는 법을 배운 겁니다.
실패가 막다른 길입니다. 루프의 부품은 다 있는데 실패한 실행이 다음 실행으로 안 이어집니다.

둘 다 모델 문제가 아니라 환경 설계 문제입니다. 모델이 강해진다고 사라지지 않아요. 강한 모델은 지름길을 더 빨리 찾을 뿐입니다.

오픈소스 Agent OS인 Ouroboros에 최근 합류한 RFC로 이 두 벽을 짚어보려고 합니다. 설계는 이슈 #1917에, 구현은 #1916에 있습니다. 전문을 직접 읽을 수 있다는 게 이 사례를 고른 이유입니다.

벽 1: 수험생에게 답안지를 쥐여줬다

대부분의 에이전트 하네스는 수용 기준을 워커 프롬프트에 그대로 렌더링합니다. 채점에 쓸 명령과 채점 기준이 될 단정문까지요.

동기는 그럴듯합니다. 에이전트가 어떻게 검사받을지 알면 과녁을 제대로 겨눌 수 있다는 거죠.

Ouroboros도 그렇게 했었습니다.
_build_success_contract_block이 verify_command와 Expected output: <단정문>을 워커 지시문에 렌더링했어요.

두 번째 누출은 더 찾기 어려웠습니다. 재시도 때 실패 사유에 단정문의 repr()이 실려서 result.error를 타고 다음 프롬프트로 되돌아왔습니다.

에이전트가 단정문을 보는 순간, 단정문을 맞추는 게 요구사항을 맞추는 것보다 싸집니다.

RFC는 이걸 직설적으로 부릅니다. 궁지에 몰린 워커의 가장 싼 경로는 수용 기준을 구현하는 게 아니라 단정 문자열을 갖고 노는 것이고, 포스트모템(seed_2be2907edc07)이 붙어 있습니다.

교과서적인 보상 해킹입니다. 능력을 본다고 생각했는데 실제로 본 건 답을 베끼는 능력이었어요.

해법: 프롬프트에 넣지 않습니다, 되돌릴 스위치도 없이

두 경로를 다 막아야 합니다. 앞문만 막으면 아무것도 안 달라집니다.

앞문: _build_success_contract_block은 이제 수용 기준 설명과 expected_artifacts만 렌더링합니다. 검증은 하네스가 따로 하니까 채점 로직이 워커의 성공 계약에 안 들어갑니다.
뒷문: 검증 게이트의 실패 사유에 단정문 repr()이 더는 안 실립니다. 재시도 힌트는 단정문을 걸러내는 전용 빌더(orchestrator/retry_hints.py)가 만듭니다. 명령 출력의 마지막 2,000자 꼬리까지요.

그 2,000자 꼬리가 베껴갈 만한 대목입니다. 앞문을 막고 로그 꼬리를 잊으면 누출은 그대로 열려 있습니다.

아직 구멍은 있어요. 지금은 문자열을 그대로 비교하는 방식이라 줄바꿈된 사본은 통과합니다. 공개 이슈(#2020)로 열어뒀습니다.

RFC에는 "공개 수준" 설정 노브가 제안됐다가 기각된 기록도 남아 있습니다. 끌 수 있는 차단벽은 누군가 일정에 쫓기는 오후에 꺼집니다. 그리고 점수가 더 좋아 보이니까 아무도 못 알아챕니다.

막힌 에이전트가 대신 받는 것

전부 숨기기만 하면 워커는 허우적댑니다. 그래서 RFC는 숨기는 것과 힌트 루프를 짝으로 묶습니다.

다음 라운드 지시문은 세션이 실제로 한 일에서 재구성됩니다. 툴 호출 궤적, 증거 목록(deliver_gate.load_ac_evidence_manifest 재사용, 읽기 전용), 검증기의 판정 결과요. 단정문에서가 아니고요.

	답을 공개하면	궤적 기반 힌트면
에이전트가 보는 것	"단정문이 요구하는 출력은 `X`"	"A와 B를 호출했고, 산출물 C가 없고, 3단계 검증에서 실패했다"
가장 싼 경로	`X`를 지어낸다	실제로 C를 만든다
점수 상승 = 능력 상승인가	아니오	예

사람 시험이 쓰는 것과 같은 배치입니다. 출제자는 답을 알고, 수험생은 자기가 어디서 틀렸는지만 배웁니다.

벽 2: 실패가 막다른 길이다

RFC에 따르면 부품은 전부 이미 있었습니다. 검증 게이트, 실행-평가 체인, evolve_step, Ralph 드라이버, focus.select_evolution_focus. 그런데 아무것도 연결이 안 돼 있었어요.

실패한 실행은 정식 평가에 안 들어갔습니다. 실패는 거기서 끝이었고 BLOCKED로 표시될 뿐이었어요.
기각된 평가는 진화에 안 들어갔습니다. 역시 거기서 끝이었고요.

루프가 끊어진 세 토막으로 있었습니다. 실패는 보고만 되고 다음 실행에 안 쓰였습니다.

해법: 실행을 평가로, 평가를 진화로 잇습니다

제약 세 개가 핵심입니다.

첫째, 실패한 실행도 평가로 이어집니다. _run_succeeded 게이트를 완화해서 세션을 만든 실행이면 정식 평가로 체인됩니다.

fail-open은 유지돼요. 큐 등록이 실패해도 실행 결과가 뒤집히진 않습니다.

둘째, 기각된 평가가 예산 있는 진화 루프를 발동합니다. 여기서 수렴 루프를 새로 구현한 사람은 없습니다. 평가 잡의 종결 경로가 final_approved is False일 때 기존 진화 기계를 큐에 넣을 뿐이에요.

새로 만든 건 1세대 다리입니다. 실행의 시드와 체인된 평가의 다중 수용 기준 체크리스트를 계보 이벤트로 옮겨서, evolve_step이 평범한 실행을 1세대로 재생하고 2세대를 이미 초점 맞춰진 상태로 시작합니다.

셋째, 실패한 수용 기준만 다음 세대로 가고 통과한 건 동결됩니다. 이 제약의 효과가 제일 큽니다.

작동시키려면 체크리스트를 ACResult로 바꾸는 변환기가 엄격한 기준을 만족해야 했습니다. 인덱스 전체 커버리지, 글자 그대로의 ac_content, semantic_ac_key 동일성이요.

이미 통과한 걸 동결하지 않는 루프는 맞게 한 일을 또 하면서 토큰을 태웁니다. 그러다 맞던 구현을 부숩니다. 증상은 세대 사이를 오가는 점수예요.

루프는 멈출 수 있어야 합니다

루프는 스스로 안 멈추니까, Ouroboros는 Ralph의 기존 정지 조건을 재사용합니다. QA 통과, 수렴, 진동 감지, 등급 퇴행, 실제 경과 시간이요.

execution.auto_evolve_max_generations의 기본값은 3이고 1~10으로 제한됩니다. BLOCKED는 예산이 소진된 뒤에만 일어납니다.

가짜 수렴을 잡는 건 진동 감지와 등급 퇴행 둘입니다. A에서 B로 다시 A로 튀는 점수, 또는 부모보다 나쁜 세대요. 둘 다 토큰을 더 태우는 대신 루프를 세웁니다.

하나 더 고쳤습니다. evolution/loop.py에 맨 except가 있어서 evaluation_summary=None으로 조용히 빠지는 경로가 세 개 있었어요. 지금은 실패 사유를 실은 기각 요약을 기록합니다.

단일 실행에서 예외를 삼키면 한 번 틀립니다. 루프에서 삼키면 오류가 N세대에 걸쳐 증폭되는 동안 로그에는 아무것도 없습니다.

싼 게이트를 비싼 게이트 앞에

같은 패턴이 설계 나머지에도 이어집니다.

평가는 층으로 나뉩니다. 기계 검증(공짜, 결정론적 검사), 그다음 의미 검증, 그다음 다중 모델 합의요. 1층에서 기각된 건 LLM 심판에게 안 갑니다.

인터뷰 단계는 층 대신 숫자를 씁니다. 얼마나 분명한지에 가중치를 매기고, 그 값을 1에서 뺀 게 모호도입니다.

Ambiguity = 1 - Sum(clarity_i * weight_i)

0.2를 넘는 점수는 스펙 생성을 막습니다. 단, force를 명시하면 통과할 수 있어요. CLI는 그 선택지를 계속·취소 옆에 띄웁니다.

루프는 마지막 두 세대 사이 온톨로지 유사도가 0.95에 도달하면 수렴합니다. 그 유사도가 진전 없이 세 세대째 평평하면 별도의 정체 감지기가 루프를 세웁니다.

README가 두 게이트의 발상을 이렇게 적습니다. 명확해지기 전에는 쓰지 말고, 안정되기 전에는 멈추지 말 것.

자기 루프에서 확인할 것

채점 기준을 수험생에게 보여주지 마세요. 하네스가 에러 메시지, 로그 꼬리, 재시도 프롬프트로 단정 문자열을 흘리는지 보세요. 앞뒤를 다 막고, 설정 옵션으로 만들지 마시고요.
실패에는 다음 단계가 있어야 합니다. 실패한 실행은 평가에 닿고, 기각된 평가는 진화에 닿고, 통과한 작업은 동결돼야 합니다.
루프는 멈춰야 하고 가짜 수렴을 감지해야 합니다. 진동, 등급 퇴행, 경과 시간, 세대 예산이요.

RFC 전문은 Q00/ouroboros#1917, 구현은 #1916입니다. 설계 문서는 트리 안 docs/hidden-checklist-convergence/에 있어요.

프로젝트는 github.com/Q00/ouroboros입니다. MIT이고, 로컬 우선이고, 13개 런타임 패밀리 앞단에서 돕니다.

직접 돌려보기

설치는 한 줄입니다.

curl -fsSL https://raw.githubusercontent.com/Q00/ouroboros/main/scripts/install.sh \
  | OUROBOROS_INSTALL_REF=devto-ko bash

파이프로 받는 게 꺼려지면 pipx install 'ouroboros-ai[mcp]'로 CLI만 받아도 됩니다. 스크립트가 더 하는 일은 MCP 서버를 호스트에 등록하는 것뿐이고, 그건 ouroboros setup으로 나중에 해도 돼요.

uv가 없으면 설치가 끝나도 MCP 서버가 안 붙습니다. 플러그인 쪽 서버를 uvx로 띄우기 때문이에요. pipx install uv나 brew install uv 아무거나 됩니다.

그다음:

ouroboros init start --orchestrator "아까 에이전트한테 시키려던 그 문장"

인터뷰가 그 문장에 남은 모호함을 0~1로 매깁니다. 0.2를 넘으면 위에서 말한 게이트에 걸리고, force도 같은 화면에 뜹니다.

재밌는 건 점수가 아니라 대답 못 한 질문입니다. 원래 안 정해두고 시작하던 게 거기서 드러나요.

한국어 문서는 ouroboros.page/learn에 있습니다. MCP에서 막히면 11장부터요.

멀티 세대 루프 돌리고 계시다면, 가짜 수렴은 어떻게 잡고 계신가요?

이 글의 영문판은 게시 후 과장 4건을 정정했습니다. 무조건 숨기는 게 아니라 글자 그대로 비교하는 것뿐이라는 것, 모호도 게이트는 force로 통과할 수 있다는 것 등이요.

이 한국어판은 그 정정이 반영된 상태로 썼습니다. 정정 이력은 영문판 말미에 있습니다.

I wrote that a stale status field is a lie with a delay on it. Then I told one.

Q00 — Mon, 10 Aug 2026 14:44:13 +0000

I wrote that a stale status field is a lie with a delay on it. Then I told one.

My last post had a section arguing that a status field nobody is forced to update becomes false by sitting still. The example was a draft marked "blocked: no route to post this." True when written. The route opened, the thing shipped, and nobody walked back to the file.

While that post sat in the publish queue, I opened a Japanese article in the same drafts folder. Its front matter said published: true. I decided that was unreliable, went to check, found nothing on the platform I expected, and concluded the article had never been posted. I wrote that up. I fixed the status field to say so. I filed a task to publish it.

The article had been live for a day and a half on a different platform.

The URL was in our own backlog, in the section for completed work. I had read that file perhaps thirty times that day. The thing that finally told me was an error message from trying to post it again: title already exists.

What each layer got wrong

The front matter was not lying. published: true is a flag the publishing tool reads, not a record that anyone published anything. It means "publish this when you run me." I read it as a claim about the past and it was a request about the future.

My correction was worse than the original. I replaced an ambiguous field with a confident, wrong one. "Never published" is a stronger claim than published: true, and I had checked exactly one platform before making it.

The checker I had written that same day for this exact class of bug did not fire, because I had given it one direction. It asks whether something claiming to be unpublished is actually live. It never asks the reverse: whether something claiming to be published actually is. The reverse is the worse failure. If we think it shipped, nobody ships it.

The smaller one, same day

The lead paragraph of that post counted seven items in a specific order: four caught by tooling, one by running the thing, one by a review bot, one by nothing at all. The body then called the review-bot item "the seventh." By the lead's own arithmetic it was the sixth.

The pre-publish check verified that the numbers summed to seven. They did. It never checked that each item sat where its own label said it sat. A number can be correct and still point at the wrong thing, which is the same failure the citation checker in that post exists to catch, one layer up.

What I would take from this

A status field describes the past. A build flag describes an intention. They look identical in YAML and they are not the same kind of statement, and I have now been caught by that in both directions in one day.

And a correction is a claim. Mine was more confident than the thing it replaced and less checked, because correcting feels like diligence and diligence feels like it has already done the verifying.

So: what is the last status field in your project that was true when someone wrote it?

Seven things I got wrong about my own project in one day, and what caught each one

Q00 — Mon, 10 Aug 2026 14:16:50 +0000

One of my checkers scans every draft I have written for wording that overstates what the code actually does. I ran it across everything, it came back clean, and I said so. At that moment three already-published posts were carrying the exact phrase it hunts for, in three languages, because all three were filed as unpublished.

That is the fourth of seven. I wrote four checkers over one day, each because the previous one had just let something through, and the interesting part turned out not to be what they found but what caught what.

Four of the seven were caught by a tool I wrote. One was caught by running the thing a tool had just approved. One was caught by someone else's review bot. The last was caught by nothing, and cost the most.

1. The docs were describing a command that had moved

The first checker was not about prose at all. It parses the CLI's typer registrations with ast and compares them against what the docs claim you can run.

It found a real defect on its first run. One line of one guide said a command was reachable only through the MCP server. It has been on the CLI for a while. Nobody noticed, because nobody had a reason to open a docs page and a registration list side by side, and the two files are 200 lines and four directories apart.

That one is now a test in CI, so the next drift fails a build instead of waiting for a reader.

2. Every merge silently invalidated my citations

The second checker is for a fact file. Every claim in it carries a path:line citation plus a probe token, and the script opens that exact line in a clone and asserts the token is on it.

It caught stale citations three times in one day. Not because anything was wrong, but because merges shift line numbers. A claim can be entirely true and still point at nothing.

I fixed the first two by hand, and on the second I guessed a line number instead of opening the file. I was off by one, so the checker failed again and I got to make the same mistake twice in five minutes. On the third I stopped fixing citations and taught the checker the difference:

the probe token is still findable in that file, at a different line, which is a stale coordinate
the probe token is gone entirely, which is a claim that may no longer hold

The first case is now a one-command rewrite. The second still fails loudly, because that is the case that actually needs a person.

Later the same day that one-command rewrite tried to corrupt the file it exists to protect.

A citation had drifted, and the fixer offered to move it from line 558 to line 255. Three hundred lines backwards is not how a merge shifts code, which is the only reason I looked. The probe token on that row was return None. It appears all over that file. The fixer takes the first match, and the first match was unrelated code three hundred lines earlier. The correct line was 559.

Had I run it, the fact file would now carry a citation pointing at code that has nothing to do with the claim, and every later check would pass, because the token really is on line 255.

The fixer now refuses when the probe appears more than once in the file and says so:

AMBIGUOUS  src/.../adapter.py:558
  probe `return None` appears 47 times in this file; first at :255
  fix: choose a probe unique to the line, then re-run.

A tool that repairs can do damage a tool that only reports cannot. The reporting version of this checker had been correct all week. The moment I taught it to write, it acquired the ability to write something false, and the thing it would have written was exactly the class of error the file was built to catch.

3. The same false claim was living in four languages

The third checker is the one that stings. It flags absolute wording about mechanisms that are real but incomplete.

What we withhold from a worker is narrower than I keep writing it, and narrower again since I wrote that sentence. When a criterion defines a verify command or an expected-output assertion, those values are omitted from the worker's contract block, so it is asked for the outcome rather than handed the assertion. Criteria are allowed to define neither, in which case there is nothing to omit. Redaction of those values anywhere else is literal over five encodings, which means an assertion that arrives line-broken through the retry path survives and reaches the worker anyway. The project's own lint and test commands are handed to it deliberately, which is a different thing and one I have described sloppily more than once. And the withholding is verbatim matching over five encodings, so an assertion that comes back reshaped, line-wrapped, diff-prefixed, or with a colour code inside it, still gets through. That is open as an issue, not a footnote.

But I had written it as "hidden unconditionally". In English. And in Chinese. And in Korean. I fixed it three times that day, in three different files, each time believing it was the last one.

Then I opened the Japanese draft and it was there too.

It survived a fourth time for a boring reason: my checker had patterns for three languages. A claim living in four languages needs rules in four languages, and mine had a hole exactly the shape of the file I had not opened yet.

4. Three published posts said things I had already corrected

This is the part I would have left out if the checkers had let me.

The checker scans drafts. It reported no violations in publishable text, and I reported that in turn. Both statements were true and both were useless, because three of those drafts were marked unpublished while the content was already live. My corrections went into files nobody was reading.

The most instructive one was a draft whose status read "blocked: no route to post this". That was accurate and honest when written. The route opened later the same day, the thread went out, and nobody walked back to the draft. An accurate failure record turned into a false status by sitting still.

So the fourth checker exists. If a draft claims it has not shipped while the log contains a live URL for that platform, it fails. It cannot tell which post came from which draft, so it prints "go look" rather than a verdict. That is the honest limit of it, and every one of that day's failures happened because nobody went and looked.

5. The file passed its schema and the command inside it was dead

This one happened after I thought I was done, and it is the cleanest version of the whole problem.

I was preparing a server.json to list our MCP server in the official registry. I fetched the published schema, ran the document through a validator, got zero errors, and wrote in my notes that it was ready to publish.

Then I ran the command the file describes, and it died.

Three things were wrong in a document a validator had just called correct. The command name was the package name rather than the console script. The runtime hint pointed at uvx, which resolves an environment the package's own extras do not pin, so the server cannot start there at all. And a required flag was missing, without which the process picks a default it cannot host.

A schema checks shape. It has no opinion about whether the thing you described exists.

Chasing the third one led somewhere I did not expect. The error I hit blamed package extras and told me to reinstall, but a clean install reproduces it, and the guard it names evaluates to false in that environment. Two different failures were sharing one message, so a user with a perfectly good install was being sent to change dependencies that were never wrong. I had already filed an issue asserting the wrong cause; I had to correct it in public and retitle it. The fix, once the cause was actually known, was to split the string in two.

I would have published a registry entry pointing at a command that cannot run, and I would have done it on the strength of a green validator.

What I would take from this if it were not my repo

Give a claim a probe, not a review. Re-reading finds a wrong sentence once. A probe finds it every time, including the times you are certain you already fixed it.

Separate a stale coordinate from a broken claim. They look identical in a diff and they need opposite responses. Conflating them is how I published a correction that was itself wrong, then had to correct the correction.

Count coverage in the dimension the content actually varies in. Mine varied by language. Yours might vary by version, by tenant, by locale. A checker is only as wide as the axis you thought of.

A status field that nobody is forced to update is a lie with a delay on it. This was the expensive one. Not the wrong sentence, but the correct record that quietly stopped being correct.

Read your own changelog before you report a bug against yourself. Every tool in this post checks whether a claim matches the source. Not one of them checks whether I have read what my project already published, and that is what cost me the most time.

Be slower to automate the fix than the check. A checker that only reports is wrong in one direction: it misses things. A checker that repairs is wrong in two, and the second one is silent.

A passing check is evidence about the check, not about the thing. Schema valid, tests green, linter clean: each of those tells you one narrow thing. The only way I found the dead command was running it. Every layer of validation I added today failed to catch something the next layer down caught by executing.

Write down how you counted, next to the count. This is the one I kept repeating. Three times in a day I published a number without saying what I had counted, and all three were wrong in the same direction, upward.

The backlog described a checker as "13 rules across 5 languages"; recounting gave 15 across 4. A note said 44 entries in a directory used a term, but that counted anything containing two words in any position; the term itself appeared 19 times. And I wrote that three competing projects package the same way as us, having checked one of them.

None of those were guesses. Each came from a real command I had run, and each was reported without the command. The number survives into the next document; the method does not, so nobody downstream can tell a measurement from an impression.

The fix is not more care. It is putting the method in the same sentence as the number, and having tools count themselves where possible. The scripts now print their own totals on every run, because a number that cannot go stale beats a number you remember to verify.

There is a sixth, and it happened while this post was queued to publish.

A review bot blocked an unrelated pull request of mine because the package summary claimed we keep the grading command out of the agent prompt. I checked: execution.context_pack defaults to on, and it appends the project's detected verify commands to the worker's system prompt. The claim was false at the package level. It was true only of the per-criterion contract, which is a much smaller thing.

That sentence had by then reached two manifests, a package summary, a registry draft, a pitch email, and a submission to a magazine with a hundred and seventy thousand stars. Section 3 of this post, the one about over-claiming, was itself over-claiming when I wrote it.

I corrected all of them and added a rule to the wording checker. Ninth instance of the same family, first one caught by someone other than me.

One last thing, and it is the one that should have been obvious.

Late in the day I filed a bug report against my own project: a dependency resolving to the wrong major version, breaking the documented install path. I reproduced it twice, read the resolver, checked a clean environment, and wrote it up carefully.

It had been fixed that morning. The fix was one flag, described in the first line of the release notes for the version I was running, which I had not read.

We publish long release notes, and I mean long: fifty-seven thousand characters in the last month. Fourteen people watch the repository. The answer to the problem I spent an hour on was sitting in a document my own project writes and almost nobody reads, including me.

None of the checkers would have caught that. There is no probe for "you did not read the thing you wrote."

The checkers are three short shell scripts and one ast walk, sitting in a marketing folder next to the drafts they police. They took less time to write than this post did. The only decision that mattered was pointing them at my own work before anyone else's, and the last thing on this list is the one none of them would have caught.

So the question I would actually like answered: what is the last thing in your project that passed a check and was still wrong?

I don't mean a bug you shipped, but something that had a green tick on it at the moment it was broken, and what the tick was measuring instead.

Your agent loop is teaching the model to cheat

Q00 — Sun, 09 Aug 2026 01:43:54 +0000

Wrapping a loop around an AI coding agent is the obvious next move once single-shot prompting stops working. Run it, score it, retry if the score is low.

Two walls show up right after:

The score goes up and the work is still wrong. The agent learned to satisfy the grader, not the task.
Failure is a dead end. Every piece of the loop exists, but a failed run never feeds the next one.

Both are environment design problems rather than model problems, and they don't go away as models get stronger. A stronger model finds the shortcut faster.

I want to walk through both using a concrete, readable example: an RFC that recently landed in Ouroboros, an open-source Agent OS. The design is written up in issue #1917, implemented in #1916. You can read the whole thing yourself, which is why I'm using it rather than describing something abstract.

Wall 1: you handed the answer key to the candidate

Most agent harnesses render the acceptance criteria straight into the worker's prompt, including the command that will grade it and the assertion it will be graded against. The motive is reasonable: if the agent knows how it will be checked, it can aim at the right target.

Ouroboros did exactly this. _build_success_contract_block rendered verify_command and Expected output: <assertion> into the worker's instructions. A second leak was harder to spot: on retry, the failure reason carried the assertion's repr(), which rode result.error back into the next prompt.

Once the agent can see the assertion, satisfying the assertion is cheaper than satisfying the requirement. The RFC names this directly: a struggling worker's cheapest path is to game the assertion string rather than implement the acceptance criterion, with a postmortem (seed_2be2907edc07) attached. That is textbook reward hacking. You think you are measuring capability; you are measuring the ability to copy an answer.

The fix: keep it out of the prompt, with no switch to turn it back on

Both leak paths have to close. Closing the forward one alone accomplishes nothing:

Forward: _build_success_contract_block now renders only the AC description and expected_artifacts. The harness verifies independently, so the grading logic does not enter the worker's prompt.
Backward: the verify-gate failure reason no longer embeds the assertion repr(). Retry hints come from a dedicated assertion-safe builder (orchestrator/retry_hints.py) that filters the assertion string out of every fragment, including the 2000-character tail of command output.

That 2000-character tail is the part worth copying. Plug the main path, forget the log tail, and the leak is still open.

The RFC also records that a "disclosure level" config knob was proposed and rejected. An information barrier that can be turned off will get turned off on some afternoon when someone is behind schedule, and nobody will notice, because the scores will look better afterward.

What a stuck agent gets instead

Hiding everything would leave the worker flailing, so the RFC pairs it with a hint loop. The next round's instructions are reconstructed from what the session actually did: the tool-call trace, the evidence manifest (reusing deliver_gate.load_ac_evidence_manifest, read-only), and the verifier's outcome. Not from the assertion.

	Disclose the answer	Trace-based hints
Agent sees	"the assertion requires output == `X`"	"you called A and B, artifact C is missing, verification failed at step 3"
Cheapest path	fabricate `X`	actually produce C
Does a rising score mean rising capability?	No	Yes

This is information asymmetry, the same arrangement human exams use: the examiner knows the answer, the candidate only learns where they went wrong.

Wall 2: failure is a dead end

From the RFC: every piece already existed (the verify gate, the run-to-eval chain, evolve_step, the Ralph driver, focus.select_evolution_focus), but nothing connected them.

A failed run never entered formal evaluation. Failure was terminal and surfaced as BLOCKED.
A rejected evaluation never entered evolution. Also terminal.

The loop existed as three disconnected segments. Failures were reported, not digested.

The fix: chain run to eval to evolve

Three constraints carry the weight:

1. Failed runs also chain into evaluation. The _run_succeeded gate is relaxed, so any run that produced a session chains into formal evaluation. Fail-open is preserved: an enqueue failure never flips the run's result.

2. A rejected evaluation triggers a budgeted evolution loop. Nobody reimplemented a convergence loop here. The evaluate job's terminal path enqueues the existing evolution machinery when final_approved is False.

The new piece is a Gen1 bridge: the run's seed plus the chained evaluation's multi-AC checklist get projected into lineage events, so evolve_step replays the plain run as Generation 1 and starts Generation 2 already focused.

3. Only failed ACs go to the next generation; passing ones freeze. This one buys the most.

Making it work required a checklist-to-ACResult converter that satisfies a strict bar: complete index coverage, verbatim ac_content, and semantic_ac_key identity. That strictness is what lets focus.select_evolution_focus freeze the passing ACs.

A loop that does not freeze what already passed will redo work it already got right, burning tokens and breaking correct implementations along the way. The symptom is a score that oscillates between generations.

The loop has to be able to stop

A loop will not stop on its own.

Ouroboros reuses Ralph's existing stop conditions: QA pass, convergence, oscillation detection, grade regression, and wall clock. execution.auto_evolve_max_generations defaults to 3, clamped to 1..10. BLOCKED only happens after the budget is exhausted.

Oscillation detection and grade regression are the two that catch false convergence: a score bouncing A to B to A to B, or a generation worse than its parent. Both halt the loop instead of burning more tokens.

One more fix. evolution/loop.py had a bare except with three silent paths to evaluation_summary=None. It now records a rejected summary carrying the failure reason, which preserves fail-closed focus semantics while making the failure durable.

Swallow an exception in a single run and you are wrong once. Swallow it in a loop and the error is amplified across N generations while your logs show nothing.

Cheap gates before expensive ones

The same pattern runs through the rest of the design.

Evaluation is tiered: Mechanical (free, deterministic checks), then Semantic, then Multi-Model Consensus. Anything rejected at layer one never reaches an LLM judge.

The interview stage uses a number instead of a tier. Ambiguity is quantified as the inverse of weighted clarity:

Ambiguity = 1 - Sum(clarity_i * weight_i)

A score above 0.2 blocks spec generation until you pass force explicitly, and the CLI puts that choice on screen next to continue and cancel. The loop converges when ontology similarity between the last two generations reaches 0.95, with a separate detector that stops it when that similarity has been flat for three generations without progress. Two mathematical gates, and the README states the idea behind both: don't write until it's clear, don't stop until it's stable.

What to check in your own loop

Don't show the grading criteria to the candidate. Audit whether your harness leaks assert strings back through error messages, log tails, or retry prompts. Close the forward and backward paths, and don't make it a config option.
Failure needs a next step. Failed runs should reach evaluation, rejected evaluations should reach evolution, and passing work should freeze.
The loop must halt and must detect false convergence. Oscillation, grade regression, wall clock, generation budget.

Full RFC: Q00/ouroboros#1917. Implementation: #1916. Design docs live in-tree under docs/hidden-checklist-convergence/ (requirements, architecture, implementation). The project is github.com/Q00/ouroboros: MIT, local-first, and it sits in front of 13 runtime families.

If you're running multi-generation agent loops in production, how do you handle false convergence? That's the part I've seen the fewest good answers to.

Update, 2026-08-10: corrected four overstatements in this post. The redaction was described as unconditional; it is verbatim matching over five encodings, so an assertion that comes back reshaped still gets through (issue #2020). The ambiguity gate was described as blocking spec generation; it can be forced past. And the convergence wording conflated two separate signals: a single similarity reading of 0.95 between the last two generations does converge the loop, while the three-generation window belongs to the stagnation detector that stops a loop making no progress. The original wording was mine and it was wrong, and so was the first version of this correction, which I published before reading evolution/convergence.py closely enough.

The AI built something reasonable. It just wasn't what I meant.

Q00 — Sun, 09 Aug 2026 00:03:30 +0000

Disclosure: I work on this project. Every number below was read out of the source, and I have included the file and line so you can check it rather than take my word for it.

Most AI coding tools fail before they write a single line of code. The prompt was vague, and the model quietly filled the gaps with assumptions you never agreed to.

You ask for "a task management CLI." The model picks a data model, a priority scheme, a persistence layer. All reasonable, none of them yours. You find out three files in, during review, and you rework it. That's the loop most of us are stuck in: prompt, guess, rework, repeat.

Ouroboros is an open-source Agent OS that fixes the input instead of the output. It's a local-first runtime layer that sits in front of Claude Code, Codex CLI, OpenCode, Gemini CLI, GitHub Copilot CLI, Kiro, Hermes, Pi, and Zcode, and replaces ad-hoc prompting with a five-stage, replayable workflow: interview, seed, execute, evaluate, evolve.

The real problem is unclear intent

Ouroboros' own framing of this is a simple table:

Problem	What happens	Ouroboros fix
Vague prompts	AI guesses, you rework	Socratic interview exposes hidden assumptions
No spec	Architecture drifts mid-build	Immutable seed spec locks intent before code
Manual QA	"Looks good" isn't verification	3-stage automated evaluation gate

The fix targets clarity, not capability.

The loop

Interview -> Seed -> Execute -> Evaluate
    ^                           |
    +---- Evolutionary Loop ----+

Interview: Socratic questioning surfaces the assumptions you didn't know you were making.
Seed: your answers crystallize into an immutable specification: acceptance criteria, ontology, constraints.
Execute: the seed runs through a Double Diamond decomposition (Discover → Define → Design → Deliver).
Evaluate: a 3-stage gate: Mechanical (free, deterministic checks) → Semantic → Multi-Model Consensus.
Evolve: the evaluation output feeds back into the next generation's seed, and the cycle repeats until the system stops learning anything new.

Each cycle is meant to converge, not just repeat. The stopping condition isn't a timer or a step count. It's math.

The interview ends when the math says so

This is the part I found most concrete. Ouroboros scores ambiguity as the inverse of weighted clarity across four dimensions (goal, constraints, success criteria, and context for existing codebases):

Ambiguity = 1 - Sum(clarity_i * weight_i)

A greenfield example from the README:

Goal:       0.9 * 0.4  = 0.36
Constraint: 0.8 * 0.3  = 0.24
Success:    0.7 * 0.3  = 0.21
                        ------
Clarity                = 0.81
Ambiguity = 1 - 0.81   = 0.19  <= 0.2 -> Ready for Seed

Above 0.2 the system keeps asking instead of letting you start on a foundation it thinks is shaky. The threshold is 0.20 at auto/interview_driver.py:125, and the weights are 0.40 / 0.30 / 0.30 at bigbang/ambiguity.py:48-50.

You can overrule it. Passing force=true bypasses the gate deliberately, and the code says so in as many words: "force=True intentionally bypasses BOTH the ambiguity threshold" (mcp/tools/authoring_handlers.py:1378). So this is a default that argues with you, not a lock. I think that is the right call, because a gate you cannot override eventually gets worked around in worse ways. But it does mean the guarantee is softer than "it will not let you."

The evolutionary loop has a matching gate on the way out. It converges when ontology similarity between the last two generations reaches 0.95, and only if that generation's evaluation was approved: high similarity with a rejected evaluation does not count (evolution/convergence.py:144, :148). A separate detector watches for stagnation: if that similarity sits unchanged for three straight generations, the loop stops as "not progressing" rather than calling it converged (convergence.py:54), and oscillation and repetitive-feedback detection run alongside so it doesn't spin on a question it already answered.

The weights are hardcoded. 40% goal, 30% constraints, 30% success criteria. That is somebody's judgment call rather than a derived constant, and it is worth knowing that before you trust the number. What I like is that it is an arbitrary you can go read and recompute, instead of one buried in a prompt.

How it actually runs

The installer auto-detects which supported runtime you're using (Claude Code, Codex CLI, GitHub Copilot CLI, OpenCode, Hermes, Gemini, Kiro CLI, Pi CLI, Zcode) and registers the MCP server where the host supports it. Everything after ooo interview (seed generation, execution, evaluation, the evolve loop) is driven from inside that same session. There's also a plain ouroboros CLI for the terminal directly (ouroboros run seed.yaml, ouroboros status executions, and so on).

ooo ralph runs the evolutionary loop across session boundaries. If your machine restarts mid-loop, it reconstructs the lineage from an event store and picks up where it left off rather than starting over.

What it does not fix

It will not make the first attempt better code. The model is the same model. What changes is that the input to that attempt is something you agreed to out loud, and there is a paper trail (the seed, the ledger, the evaluation stages) to read afterward instead of reconstructing what you meant from a diff.

It also costs you the thing some people came for. If your actual want is to type one line and walk away, an interview that keeps asking is friction, and calling that friction a feature does not make it stop being friction. The bet is that the questions are cheaper than the rework. That bet is wrong for a throwaway script and I would not use it there.

Try the part that takes 30 seconds

You do not have to install anything to judge the idea. Open the scoring code and see whether you agree with the weights:

bigbang/ambiguity.py:48-50: 40% goal, 30% constraints, 30% success criteria.

If you want to run it:

curl -fsSL https://raw.githubusercontent.com/Q00/ouroboros/main/scripts/install.sh | OUROBOROS_INSTALL_REF=devto bash

Then, inside your agent session, run these in order (ooo setup is a one-time step):

> ooo setup
> ooo interview "I want to build a task management CLI"

The interview is the whole pitch. If it asks you something you had not decided yet, that is the product working. If it asks you three things you had already written in the prompt, that is a bug and I would like to see the transcript.

MIT, Python 3.12+, runtime guides per CLI: github.com/Q00/ouroboros.

The question I actually want answered

Everybody agrees vague prompts are the problem. Almost nobody agrees on who should fix it. Three positions I keep running into:

The model should ask. Clarification is the model's job and a harness that does it is a workaround for a weak model.
The harness should ask, because you want the same questions every time regardless of which model is behind it.
Neither should ask. Write a better prompt.

I built around position 2 and the ambiguity score is what that position looks like in code. If you hold 1 or 3, I would rather hear the argument than the star.

Where do you catch it today? During the interview, at review, or three files in?