Test AI Agent Self-Correction With an Independent Oracle, Not Confidence

“Self-correcting” sounds reliable until the same model decides whether its own revision is correct. More loops can produce more confidence without producing better code.

Build the correction path around an independent oracle:

candidate v1
  → deterministic tests fail
  → revision cites failure evidence
  → candidate v2
  → independent tests pass
  → policy checks pass
  → human review or bounded acceptance

Version every input:

{
  "task": "fixture-17",
  "candidate": 2,
  "replaces": 1,
  "oracle_version": "tests@a8f12c",
  "failure_ids": ["rounding-negative-3"],
  "status": "pending"
}

Then inject unreliable evidence. Make one test flaky, one oracle stale, and one tool falsely report success. The system should abstain or request review when evidence conflicts; it must not count another model response as independent verification.

Core invariants:

• A revision identifies the failure it intends to fix.
• Acceptance references a versioned oracle.
• A changed oracle invalidates cached acceptance.
• Retry count is never reported as correction quality.
• The denominator includes abstentions and false corrections.

The open-source OpenAI Evals repository provides examples and infrastructure for evaluating model behavior. For coding agents, repository tests and policy checks can form a more task-specific oracle, provided their own limitations are recorded.

MonkeyCode publicly supports a multi-model workflow and has a cloud platform currently free to start. That makes a versioned fixture useful for external comparison, but I am not publishing an unrun model ranking or claiming a MonkeyCode reliability result.

I am a MonkeyCode user, not affiliated with the project. This account shares an operator with the other accounts in this batch.