DEV Community

Finley Zhou
Finley Zhou

Posted on

Admit an Agent Patch Only When Oracle, Fixture, and Freeze Ledgers Agree

An agent patch does not earn a score because a property command exited zero. It earns a score only when three ledgers agree: an oracle ledger written without the diff, a fixture ledger pinned to an epoch and a digest, and a flake-freeze ledger bound to a seed, a base revision, and an expiry. If any ledger is missing or inconsistent, the gate should refuse. A refusal is a result. A bare pass is not.

That rule is stricter than running the tests, and narrower than a proof. It blocks three false credits that show up in agent-patch review. An oracle can restate the patch. A fixture can drift between runs. A freeze can hide a regression without a bound.

What each ledger is allowed to claim

The oracle ledger claims that a property was selected from a specification, a bug report, or an invariant that existed before the candidate diff. It does not claim the property is complete. It claims authorship order, which is the part a green run cannot show.

The fixture ledger claims that the base run and the candidate run saw the same epoch. Epoch means a named snapshot: repository revision, fixture file digest, and runner image id. A digest match without an epoch pin is weaker than it looks, because two snapshots can share a copied file and still differ in process state.

The freeze ledger claims that a named seed was unstable on the base revision, and that the quarantine expires. It does not claim the property is unimportant. It claims a temporary exclusion from the score, plus a replay obligation on that same base.

Oracles define what must hold. Fixtures define what was held constant. Freezes define what was excluded, and until when. Those three claims are not interchangeable, so a single log line cannot substitute for a missing ledger.

Step 1: Author the oracle before opening the diff

Read the specification or the failing issue first. Write properties against that text, and hash or commit them before the patch file is in the worktree. Opening the diff first makes independence a retrospective label.

A useful property names inputs, a relation, and a failure witness. A weak property returns true for every input the generator can draw. Keep weak properties out of the score, even when they remain useful as smoke checks.

# Proposal only. Not an executed measurement.
def balance_never_negative(account, ops):
    balance = account.opening
    for op in ops:
        balance = apply_op(balance, op)
        if balance < 0:
            return False
    return True
Enter fullscreen mode Exit fullscreen mode

Store the oracle as data, not as a comment beside the patch. The gate reads fields. It does not read intent from a pull-request description.

{
  "oracle_id": "bal-nonneg-01",
  "source": "spec/accounts.md#invariants",
  "authored_commit": "base-or-earlier",
  "sees_diff": false,
  "property": "balance_never_negative"
}
Enter fullscreen mode Exit fullscreen mode

If sees_diff is true, refuse with ORACLE_NOT_INDEPENDENT. A model may draft the wording. A reviewer may clear the flag only after checking the draft against the spec, not against the hunks. The field is a process control, not a proof that nobody peeked.

Step 2: Pin the fixture epoch and publish the digest

Compute the digest from the fixture bytes the runner will load. Do not hash a directory listing and call that fixture identity. Identity has to be the payload, plus the image that loads it.

# Illustrative commands. Confirm paths against your tree before use.
sha256sum fixtures/accounts.json > ledger/fixture.digest
git rev-parse HEAD > ledger/base.rev
printf '%s\n' "$RUNNER_IMAGE_ID" > ledger/runner.image
Enter fullscreen mode Exit fullscreen mode

Fail closed when any required field is empty. An absent image id is not a default value. It is an unpinned epoch, and unpinned epochs do not get a score.

# Proposal only. Unexecuted.
REQUIRED_FIXTURE_FIELDS = (
    "epoch_id",
    "base_rev",
    "digest_sha256",
    "runner_image_id",
)

def fixture_refusal(entry):
    missing = [k for k in REQUIRED_FIXTURE_FIELDS if not entry.get(k)]
    if missing:
        return ("FIXTURE_EPOCH_UNPINNED", missing)
    if entry.get("observed_digest") != entry["digest_sha256"]:
        return ("FIXTURE_DIGEST_MISMATCH", entry.get("observed_digest"))
    return None
Enter fullscreen mode Exit fullscreen mode

Run the base and the candidate only after this ledger exists. If the candidate rewrites fixture format, stop and review that migration alone. Updating the digest in the same commit as the format change hides the drift this file exists to catch.

Step 3: Separate stable base failures from frozen seeds

Run the oracle on the base revision before the patch is applied. Record the seed for every hard failure and every retry disagreement. Without that split, quarantine becomes a bucket for every red test.

A stable base failure is not a flake. Do not freeze it. Either the oracle is wrong, or the base is wrong, and freezing it would launder a known defect into the candidate score.

A flake freeze is valid only when every field in the table is present and internally consistent. Absence maps to a refusal, not to a default pass.

Field Required meaning Refusal if absent or inconsistent
property_id Oracle being excluded from the score FREEZE_UNBOUND
seed Input that disagreed across retries FREEZE_UNBOUND
base_rev Revision where instability was observed FREEZE_REPLAY_NOT_ON_BASE
fixture_epoch Epoch of that observation FIXTURE_EPOCH_UNPINNED
observed_on Must be base until a later review FREEZE_REPLAY_NOT_ON_BASE
expires_at Date after which the exclusion is void FREEZE_EXPIRED
owner Person or ticket accountable for removal FREEZE_UNBOUND
{
  "property_id": "bal-nonneg-01",
  "seed": "0x9c1e",
  "base_rev": "a11ce",
  "fixture_epoch": "epoch-3",
  "observed_on": "base",
  "expires_at": "2026-10-10",
  "owner": "patch-review-queue"
}
Enter fullscreen mode Exit fullscreen mode

The dates and ids in that object are placeholders, not a recommended retention period and not a measured flake rate. Pick an expiry the team can actually honor. A freeze without an expiry is a permanent waiver, and this gate treats permanent waivers as refusals.

Step 4: Replay on a fresh runner, then score the complement

Order is part of the result. A different order can produce the same console text and a different claim. Follow the sequence below, and keep the intermediate records.

  1. Rebuild the runner from the pinned image. Do not reuse a workspace that already applied the patch.
  2. Replay each frozen seed on the base revision. Confirm the recorded instability still matches the freeze.
  3. Apply the candidate on a fresh tree at the same fixture epoch.
  4. Run those frozen seeds on the candidate. A new hard failure, outside the recorded instability, is a regression.
  5. Run every property that is not frozen. Only that complement may enter the score.
  6. Emit admit or refuse, and always include the refusal list. An empty list is meaningful. A missing list is not.
# Proposal only. Unexecuted. Pure decision helper.
from datetime import date

def admit(oracle, fixture, freezes, today, score_inputs):
    refusals = []
    if oracle.get("sees_diff"):
        refusals.append("ORACLE_NOT_INDEPENDENT")
    fixture_problem = fixture_refusal(fixture)
    if fixture_problem:
        refusals.append(fixture_problem[0])
    for freeze in freezes:
        bound = all(
            freeze.get(k)
            for k in ("property_id", "seed", "owner", "expires_at")
        )
        if not bound:
            refusals.append("FREEZE_UNBOUND")
            continue
        if freeze.get("fixture_epoch") != fixture.get("epoch_id"):
            refusals.append("FIXTURE_EPOCH_UNPINNED")
        same_base = (
            freeze.get("observed_on") == "base"
            and freeze.get("base_rev") == fixture.get("base_rev")
        )
        if not same_base:
            refusals.append("FREEZE_REPLAY_NOT_ON_BASE")
        try:
            expiry = date.fromisoformat(freeze["expires_at"])
        except (TypeError, ValueError):
            refusals.append("FREEZE_UNBOUND")
            continue
        if expiry < today:
            refusals.append("FREEZE_EXPIRED")
    frozen_ids = {f.get("property_id") for f in freezes}
    scored = [p for p in score_inputs if p.get("property_id") not in frozen_ids]
    if not scored:
        refusals.append("NO_UNFROZEN_PROPERTIES")
    elif any(p.get("status") != "pass" for p in scored):
        refusals.append("UNFROZEN_PROPERTY_FAILED")
    decision = "admit" if not refusals else "refuse"
    return {
        "decision": decision,
        "refusals": sorted(set(refusals)),
        "scored": len(scored),
    }
Enter fullscreen mode Exit fullscreen mode

A patch that passes only frozen properties is a refusal, via NO_UNFROZEN_PROPERTIES, not a quiet success. That code is the backstop when quarantine grows until the suite is empty. Count the complement, or do not publish a score.

The helper treats an expiry equal to today as still valid, and it does not interpret time zones. Invalid date strings fail closed as FREEZE_UNBOUND. A proposed unit test, not a recorded run, locks the empty-complement case:

# Proposal only. Not executed in this article.
def test_empty_complement_refuses():
    decision = admit(
        oracle={"sees_diff": False},
        fixture={
            "epoch_id": "epoch-3",
            "base_rev": "a11ce",
            "digest_sha256": "abc",
            "runner_image_id": "img-1",
            "observed_digest": "abc",
        },
        freezes=[{
            "property_id": "bal-nonneg-01",
            "seed": "0x9c1e",
            "owner": "queue",
            "expires_at": "2026-10-10",
            "observed_on": "base",
            "base_rev": "a11ce",
            "fixture_epoch": "epoch-3",
        }],
        today=date(2026, 9, 26),
        score_inputs=[{"property_id": "bal-nonneg-01", "status": "pass"}],
    )
    assert decision["decision"] == "refuse"
    assert "NO_UNFROZEN_PROPERTIES" in decision["refusals"]
Enter fullscreen mode Exit fullscreen mode

Swap in one unfrozen passing property, clear the freeze list, and the same helper returns admit. That contrast is the behavior worth pinning in review. It is not evidence about any live suite.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access fits one narrow step: drafting candidate property text from a specification excerpt, before the diff is opened. Treat that draft as untrusted ledger input. A reviewer sets sees_diff to false only after the draft cites the spec, names a witness, and does not mention patch hunks. The model does not admit the patch. The admit helper does, and only from the ledgers you reviewed.

MonkeyCode's free server option fits step 4 as an ephemeral runner. Start from a clean image, load the pinned fixture, replay the base, then run the candidate. Discard the machine afterward so a later epoch cannot inherit packages, caches, or a half-applied diff. A free tier is not a contract. Do not encode it as a required CI dependency, and do not assume a fixed quota, hardware shape, region, or retention period. Those details are not established here.

If either option is unavailable, the ledgers still run on any runner you control. The admission rule does not depend on a vendor path. The vendor path is only a way to draft the oracle file and to rent a clean machine for the replay.

Limitations

Independence is only as honest as sees_diff. Nothing in the helper detects a reviewer who read the diff and flipped the flag. For a stronger control, require the oracle commit to precede the patch commit, and reject pairs that land in the same commit.

Digest checks do not see mutable state outside the hashed files. Clocks, network stand-ins, and unpinned package indexes can still fork two runs that share a digest. Pin those inputs, or write them down as residual risk on the fixture ledger.

A freeze replay that still flakes does not prove the candidate preserved behavior. It proves the exclusion still refers to the recorded seed. Behavior outside that seed is what the unfrozen score has to cover, which is why an empty complement refuses.

This protocol does not rank patch quality, estimate defect rates, or replace security review. It answers a narrower question: is this property score admissible? The sample code is a proposal. It has not been executed against a production suite here, and it omits parallelism and partial file reads. Unit-test it on synthetic ledgers before it can block a merge.

Who should skip this gate

Skip it when there is no rebuildable base revision. A freeze cannot be bound to a base you cannot replay, and an unbound freeze is already a refusal.

Skip it when the change is a fixture migration. Land the migration under its own review, then open a new epoch. Same-commit digest updates are how fixture drift gets signed off by accident.

Skip it when the runner is shared and stateful. A reused machine breaks epoch isolation, including a free-server session that is not discarded between candidates. Fresh image, or no claim about the fixture.

Skip it if expired freezes will not be deleted. A ledger nobody expires becomes a second skip list, quieter than a marked skip and harder to grep for during review.

What to put in the pull request

Ship four artifacts with the patch, not a screenshot of a green job. Reviewers should be able to recompute the decision without opening a dashboard.

  1. ledger/oracle.json, with sees_diff set and a spec pointer.
  2. ledger/fixture.json, with epoch, base revision, digest, and runner image id.
  3. ledger/freezes.json, either populated or an explicit empty list.
  4. The admission output: admit or refuse, plus the refusal codes.

Read the refusal codes before the diff size. A short patch with ORACLE_NOT_INDEPENDENT is not ready for a score. A long patch with an empty refusal list and at least one unfrozen pass is eligible for the rest of review. It is not exempt from that review.

If those ledger files already live in git, draft the oracle from the spec with MonkeyCode's free model access and run the replay on its free server. Keep admit in code you review before either artifact can block a merge.

Top comments (0)