DEV Community

Cover image for Three Agents Returned Three Reports. Which Task Did They Actually Prove?
joinwell52
joinwell52

Posted on Originally published at joinwell52-ai.github.io

Three Agents Returned Three Reports. Which Task Did They Actually Prove?

Original research: English · 中文 · Research Center

Three Agents finish at nearly the same time. Each report says “tests passed.” The dashboard sees three successful files. Yet one report came from the wrong project root, one reused an older test log, and one belongs to a retry that had already been rejected.

Three green reports can still add up to a false delivery.

Acceptance needs three plain answers:

  1. Who did the work? Runtime identity and immutable execution context.
  2. What proves it? Machine-captured command, exit code, Git state, logs, and artifact hashes.
  3. Which task owns it? A unique task contract plus the accepted execution attempt.

Matching filenames alone cannot answer all three.

The attribution chain 

task dispatch
→ immutable context binding
→ tool execution
→ machine evidence capture
→ report envelope
→ attribution validation
→ independent QA
→ acceptance decision
Enter fullscreen mode Exit fullscreen mode

The report is one link. It is not the whole proof.

Execution facts and independent verification bind to Task 42 while Task 17 is rejected

Figure 1. Evidence is accepted only when identity, task binding, execution facts, and independent verification converge on the same task.

Do not let the model author the envelope

A common file protocol checks that the task number inferred from the filename equals frontmatter.task_id and the first referenced task. This catches many accidental mismatches.

But an LLM can copy an old template and put the same wrong task ID in all three places. The file is internally consistent and externally wrong.

The stronger design is Runtime packaging:

  1. the Agent submits report content through a controlled tool;
  2. the Runtime injects task_id, parent_id, project root, runtime instance, attempt ID, and evidence references from the active execution context;
  3. the Agent cannot overwrite those envelope fields;
  4. the validator compares the envelope with the dispatch record and current task contract.

Not every current submission path implements universal envelope injection and unified evidence capture. They are target architecture requirements. The article does not present them as already complete everywhere.

Evidence must come from the execution boundary

“I ran the tests” is a claim. Useful evidence includes:

  • exact command and normalized working directory;
  • process exit code;
  • captured stdout/stderr or a content hash and durable location;
  • Git commit/tree or changed-file hashes;
  • Runtime instance and execution attempt;
  • tool invocation identity;
  • test time and environment facts when they affect interpretation.

The model may summarize this evidence, but it should not fabricate the primary record. A Runtime or tool wrapper should capture it at the process boundary.

Use logical order, not filesystem time

File modification time is unreliable for causality. Git checkout, archive extraction, clock skew, and cross-device synchronization can rewrite or preserve timestamps in surprising ways.

Use a task revision and attempt sequence:

TASK-42 revision 6
  attempt 1 → rejected
  attempt 2 → accepted evidence candidate
Enter fullscreen mode Exit fullscreen mode

If a late report from attempt 1 arrives after attempt 2, physical time does not make it current. The accepted task revision and attempt decision do.

Multiple reports need an explicit state machine

Never overwrite a report silently. Each submission should be immutable and identified by attempt or report sequence. The acceptance layer can then decide:

  • candidate: awaiting verification;
  • rejected: evidence or contract mismatch;
  • superseded: replaced by a later valid attempt;
  • accepted: bound to the current task decision.

History remains inspectable, while only one attempt may satisfy the task.

A practical attribution veto list

Reject or hold acceptance when:

  • Runtime instance, canonical root, task revision, or attempt conflicts;
  • a report refers to a task other than the active dispatch contract;
  • Git hash or artifact hash does not match the inspected workspace;
  • required machine evidence is missing or cannot be resolved;
  • the execution right expired on a path that implements a lease;
  • a rejected/superseded attempt submits a late report;
  • independent verification covers a different revision;
  • three metadata fields agree with one another but disagree with the Runtime dispatch record.

A veto is not automatically an accusation of fraud. The neutral verdict is stale, conflicting, or insufficient evidence unless there is independent evidence of deliberate deception.

What 44 tests establish

On the pinned CodeFlowMu implementation, 44/44 report ownership and aggregation gate tests passed. They validate the covered parser, ownership, and aggregation paths.

They do not prove report content is inherently truthful. They also do not cover the logical counterexample where all three model-authored fields consistently point to the same wrong task. That case remains a required next fault test and a reason to move envelope authority into the Runtime.

The durable rule is: a report counts only when identity, machine evidence, task contract, and an authorized acceptance decision agree.

The full fact matrix and implementation boundaries are in the canonical English article. A Chinese version is also available.

More bilingual Agent engineering research: JoinWell52 Research Center

Top comments (0)