When an independent audit corrected a coding-agent test report

Necip Reis — Wed, 07 Oct 2026 23:19:39 +0000

I am the founder of AiOrch, the commercially licensed engineering orchestration platform discussed here.

A coding-agent workflow can produce several different kinds of evidence: an implementation, a review verdict, an integrated branch, a test result and a pull request. Those records can disagree.

In a recorded AiOrch Python TODO CLI demo, the final development report claimed 36 tests. A separate audit reran python -m pytest tests/ -v and recorded 35 passed with exit status zero.

That is a small discrepancy. It is also a useful example of why the generated report needs its own verification boundary.

What the run actually recorded

The work was assigned to three implementation agents covering core logic, CLI work and packaging. Dependent work started after its prerequisite was approved. All three implementations received change requests before their final approval. The changes addressed scope cleanup, CLI error handling and packaging/documentation corrections.

During integration, the record captured a conflict on the CLI branch and its later resolution. All three branches were marked merged inside the session.

A separate audit session used four audit/follow-up agents. In addition to correcting the test count, it identified specification/report discrepancies and prepared follow-up fixes. The inspected record did not show a merged audit result. An approved audit session therefore does not establish that all of its follow-up changes reached the public PR.

The case study links to demo-projects PR #5. It was still open when checked on 5 October 2026. Branch integration inside the session and a GitHub PR merge are different states.

Five checks worth adding to an agent handoff

Preserve the approved task specification. In this historical run, an original task file was removed and the audit reconstructed its specification from saved configuration. A durable copy gives later reviewers a stable reference.
Record the exact command and exit status for each claimed validation result. A successful result without a command, revision and captured output is difficult to reproduce.
Distinguish evidence from the agent's interpretation of it. Keep test output separately from the prose summary so a count or failure classification can be checked.
Show integration state explicitly. A reviewed branch, an integrated branch and a delivered PR should each have their own status. If audit fixes have not been integrated, say so.
Preserve unresolved gates. Missing integration evidence should remain visible in the handoff instead of being silently translated into a pass.

Why parallelism changes the handoff

Separate git worktrees make it possible to assign independent implementation work without agents overwriting a shared checkout. They do not prove that the combined result satisfies the task. Dependencies, changed interfaces and merge conflicts still require integration work.

Review verdicts and test commands answer different questions. A reviewer can identify scope or behavior problems; a test command can establish what passed for a particular checkout. Keeping both records makes the final result easier to inspect.

The scope of this example

This is a small recorded Python demo, not a tool-comparison benchmark or a claim of production reliability. The run contains resume events, and the record does not identify the actor for every resume. It does not establish a run without operator intervention.

Usage data was missing for several agents, so the dashboard's displayed zero cost cannot support a claim that the run was free. The published case study inspected historical results; it did not rerun them during publication.

AiOrch runs its orchestration and worktrees on your infrastructure. Configured cloud inference providers still receive the context sent to them. Self-hosting the coordinator should not be described as proof that all inference remains local.

The useful engineering result here is an inspectable chain from task to implementation, review, integration, audit and handoff—with disagreements retained in the record.

Full case record, limits and public PR

Parallel orchestration guide

DEV Community: Necip Reis

When an independent audit corrected a coding-agent test report

What the run actually recorded

Five checks worth adding to an agent handoff

Why parallelism changes the handoff

The scope of this example