Your agent loop can be wrong and still report success. After implementing a change and running its checks, the implementation agent marks the task complete. A downstream agent can trust that state and keep building in another worktree, so the failure can stay hidden until integration or production reveals that several changes depend on the same mistake.
You create a verification gap when you keep loop engineering's automation and parallelism but drop the independent check that Addy Osmani's framework treats as structural. If the implementation agent also decides when the work is done, its completion claim becomes your harness's stop signal.
A verification loop breaks that chain by evaluating the artifact against a done condition the implementation agent cannot change and blocking the harness until independent evidence supports completion.
Multi-agent loops scale unverified output
When you add parallel agents, implementation throughput can outgrow your ability to validate their changes.
Peter Steinberger's reported month of agent activity shows how quickly output can scale. His API dashboard showed $1,305,088.81 in OpenAI usage over 30 days, covering 603 billion tokens across 7.6 million requests and roughly 100 Codex instances.
Those figures measure activity and token cost, while correctness requires its own evidence.
Uber's response addressed the cost side of the same problem. The company reportedly used its 2026 AI budget in about four months, then capped employees at $1,500 per month for each AI coding tool. That token-cost budget set a financial boundary after usage scaled without measuring output quality.
AgentField shows the corresponding quality failure in code. In its first run with more than 30 parallel harnesses, a pull request looked correct and every test passed. One agent had built an API layer on a module that another agent never exported. The downstream agent mocked the dependency, so the real boundary never entered the test path.
Those tests were locally valid and globally misleading because the failure appeared only after the components were assembled. An integration check that built the upstream package and consumed its public API from a real downstream consumer would have exposed it.
Parallel worktrees isolate file changes while leaving cross-worktree assumptions unchecked. Each agent can satisfy its local task while the assembled system remains broken, so verification has to run across the dependency boundary that connects the artifacts.
Review beyond local tests with an AI code review checklist that catches hidden assumptions and production constraints.
How a verification loop should work
Your verification loop should start with a read-only acceptance policy that the implementation agent cannot rewrite. The verification stage runs outside the implementation agent's process. When a check requires judgment, a verifier agent reviews the immutable change against that policy. It receives the evidence in an isolated context and returns a verdict.
The agent harness applies that verdict to the loop's next state. It retries the task, waits for an external condition, escalates to the release owner, or records completion. This control code can live in a workflow engine or an ordinary function around a while loop.
In a plan-execute-verify loop, the done condition names an observable result before execution begins, such as a passing consumer integration test or a public export.
Keep the verification contract and test runner outside the implementation agent's write scope. They can live in protected CI or harness-owned configuration as long as the implementation agent has read-only access to the acceptance criteria and cannot alter the evaluator.
Give the verifier agent the task specification, an immutable artifact, the pinned dependency state, and raw results from the required checks. Its context excludes the implementation agent's reasoning and completion claim, leaving raw evidence from the artifact to determine its verdict.
For an SDK change, the verification contract could look like this.
task: Expose AuthClient through the public SDK
artifact: commit 8f31a2
done_condition:
- The public package exports AuthClient
- A consumer can import and instantiate AuthClient
- Existing SDK tests still pass
required_evidence:
- Package build output
- Public export list
- Consumer integration test
allowed_verdicts:
- YES
- NO
- MAYBE
- IFF
After the implementation agent produces a candidate change, the agent harness resolves artifact to an immutable commit or build. It runs the required checks against the pinned dependency state and attaches the raw results. The verifier agent compares that evidence with each item in done_condition, rejecting a local mock as proof that an external dependency exists.
Binary pass or fail loses unresolved evidence and conditional completion, which the four-value model preserves.
| Value | Meaning | Loop action | Example trigger |
|---|---|---|---|
| NO | Evidence disproves at least one predefined condition. | Return the failed condition to the implementation agent. | A failing test or absent export. |
| YES | All predefined conditions have supporting evidence. | Record completion and end the agent loop. | Tests pass, and the required export is present. |
| MAYBE | The available evidence cannot settle the decision, or human judgment is required. | Pause and send the evidence to the release owner. | Automated checks cannot resolve the security impact. |
| IFF | Local conditions passed, and global completion is equivalent to a named external condition. | Keep the loop blocked. Convert to YES when the condition becomes true or NO when it fails definitively. | The loop is complete if and only if the required upstream version is published. |
IFF means "if and only if" and expresses a strict dependency in which the loop may return YES only when the named external condition becomes true. Evidence conclusively covers the artifact's local requirements while one external invariant still controls global completion. This conditional verdict is the four-value model's new contribution.
A token limit can stop a run without proving completion, which keeps resource control separate from termination logic.
Persist each verdict outside the model context with its artifact ID and evidence references while pinning the exact verification-policy version to the same record. The record must survive process restarts so a MAYBE verdict remains paused and a IFF task resumes when its named condition changes. Repository state or a workflow database can hold this record.
When the verifier returns NO, the harness sends the failed condition and raw evidence back to the implementation agent. The next iteration starts from that recorded evidence and the pinned policy while the verifier's reasoning remains outside the implementation context.
Four rules keep verification independent as the loop scales.
Use independent evidence because separate context windows can still preserve the same mistake when both agents rely on generated tests or stale dependency state.
Run deterministic checks first, then call a verifier agent only when the criterion requires judgment that an executable check cannot provide.
Test the assembled system from a clean checkout or deployed preview when work spans packages or services.
Match the evidence threshold to the release risk by using link and example checks for documentation and consumer integration tests for public APIs, while security-sensitive releases and external side effects require named human approval.
The human owns the production decision
Human-in-the-loop architecture places a named release owner at the production boundary while the agent harness automates implementation and verification under that policy. A YES verdict ends the agent loop, after which low-risk changes may advance automatically while higher-risk changes require explicit approval.
The release owner remains accountable when an agent executes a production action. When automated evidence no longer settles the operational risk, MAYBE routes the evidence to that owner for a decision.
The evidence package for a required review begins with the immutable artifact and original verification contract. Its verification verdict is attached to raw test output so the release owner can inspect the evidence directly. Unresolved assumptions appear beside the rollback and ownership details.
Forward Deployed Engineering puts this production boundary into practice by having engineers validate production readiness with the team that will run the system. Applied to agent workflows, this keeps production policy with the people responsible for operating the result.
Osmani describes that production boundary as the handoff between the agent's inner execution loop and the human-owned outer loop, where people retain goal setting and production risk.
Add verification before you scale
Give one production-facing loop an independent rejection path before you add another agent or worktree.
Convert its acceptance criteria into a stored verification contract, then map every named check and allowed verdict to a harness action.
Run the verification stage from a clean environment against real outputs for one sprint without letting its verdicts control the loop. In this observation period, it records what it would decide while your existing process determines what advances.
Record the implementation agent's completion claim and verification verdict alongside the release owner's adjudication and the production outcome.
Define the false-
YESand escalation thresholds the verification stage must meet before it receives authority over task termination, then name the release owner responsible for the production policy and for adjudicating every disagreement during the trial.Grant the verification stage authority over task termination only after it meets both thresholds, and revise the policy before repeating the observation period if it misses either one.
Classify each outcome as an incorrect completion claim, a verifier false YES, a false rejection, or an unresolved case that remains blocked as a MAYBE escalation and stays out of verifier accuracy calculations until adjudication.
| Metric | Calculation | What it reveals |
|---|---|---|
| Disagreement rate | Agent-verifier disagreements divided by outputs checked | Path divergence. |
| Confirmed miss rate | Incorrect agent completion claims divided by adjudicated completion claims | Accuracy of agent completion claims. |
| Verifier false-YES rate | Incorrect YES verdicts divided by adjudicated YES verdicts | Precision of verifier approvals. |
| False-rejection rate | Valid outputs rejected divided by all adjudicated valid outputs | Rate at which valid work is blocked. |
| Escalation rate | MAYBE verdicts divided by outputs checked | Demand for human adjudication. |
Accuracy rates require an adjudicated outcome from human review or production evidence. Disagreement and escalation can be measured directly, while verifier accuracy still depends on adjudication.
Top comments (2)
The line that lands hardest here is "if the implementation agent also decides when the work is done, its completion claim becomes your harness's stop signal." We hit exactly that running parallel agents, and the subtle part is that "independent" has to mean more than "a second agent." If your verifier shares the same model, the same context window, and the same success framing as the implementer, it tends to rationalize the same blind spot — you get agreement, not verification. What actually broke the chain for us was making the done-condition external and mechanical wherever possible: a check the agent can run but can't author or edit, in a fresh context that never saw the implementation reasoning. Curious how you draw the line on independence — do you isolate the verifier's context, vary the model, or both? And do you let it emit only pass/fail against a fixed spec, or can it renegotiate what "done" means? That last freedom is where I've watched verification quietly collapse back into self-grading.
The failure mode I keep seeing is the verifier sharing the same happy path as the builder. A second agent helps only if it has a colder contract to check against, like a clean checkout, real integration build, and a tiny list of claims it is allowed to reject. Otherwise it mostly produces a more confident green light.