Pin the Oracle Before the First Agent Iteration

A common Friday incident looks like this. An agent loop is aimed at a flaky integration test. The model edits three files, reruns a subset of tests, and prints that everything is green. CI on main is still red.

The transcript never defined what green meant. The loop had a budget. It did not have an oracle.

Free model access makes that pattern cheap to repeat. A free server makes residue easy to leave behind. Neither one tells you when to stop.

What this article is

This is a glossary, a four-question tree, and a worked example at each leaf. The artifact is a tiny oracle file plus a wrapper that refuses to start the loop if the oracle is missing.

The method does not depend on a vendor. Strip the product note later and the tree still holds.

Glossary

Use these terms as written. Do not substitute model confidence for any of them.

Oracle. A check outside the model's self-report that can accept or reject the loop's output. A test command, a typecheck, a golden file, or a written human rubric can be an oracle. A chat reply that says "done" is not.

Budget envelope. The maximum iterations, wall-clock time, and disk the loop may consume. A budget without an oracle only limits damage. It does not define success.

Loop residue. Files, processes, virtualenvs, and logs left on the machine after the loop stops. Residue is evidence. It is not a merge.

Promotion gate. The step that may copy residue onto a real branch: human review, CI on a pull request, or both.

Sketch loop. A loop whose residue must be wiped. Useful for exploration. Unsafe as a source of production diffs.

Bound loop. A loop that may produce a candidate patch only because an oracle can fail it.

The tree below branches on oracle class. It does not branch on tone, latency, or how sure the model sounds.

The four-leaf tree

Work the questions in order. Stop at the first leaf that fits.

1. Can you write a command that is red on the current tree and that you will trust if it turns green?
2. If not: can a human reject the result in a fixed timebox using a written rubric of five lines or fewer?
3. If not: is the session explicitly a sketch, with a wipe command and no path onto main?
4. If not: do not start the loop.

The leaves are Automated, Rubric, Sketch, and Refuse.

Leaf A — Automated oracle

When. You already have, or can add in minutes, a deterministic check: pytest, go test, npm test, cargo test, a linter that fails on the defect, or a contract test against a fixture.

Worked example. Ticket: "CSV export prepends a BOM that Excel-on-macOS treats as part of the header."

Proposed oracle, labeled illustrative and not executed here:

# oracle.sh — must fail on HEAD before the loop is allowed to start
set -euo pipefail
python -m pytest -q tests/test_csv_export.py

# tests/test_csv_export.py
def test_csv_header_has_no_bom(tmp_path):
    from app.export import write_csv
    out = tmp_path / "out.csv"
    write_csv(out, rows=[{"name": "Ada"}])
    raw = out.read_bytes()
    assert not raw.startswith(b"\xef\xbb\xbf")
    assert raw.splitlines()[0] == b"name"

Numbered workflow:

1. Commit or stash so residue is diffable.
2. Run the oracle. Confirm it is red. If it is already green, you do not have an oracle for this ticket.
3. Start the agent only against that command and an allowlist of paths.
4. Stop when the oracle is green, the budget envelope is hit, or the model edits a file outside the allowlist.
5. Promote only through a pull request. A green oracle is necessary. It is not sufficient.

If step 2 is green on HEAD, stop. You are about to optimize a passing test.

Leaf B — Rubric oracle

When. The defect is visual, editorial, or judgment-heavy. No cheap automated check exists yet. A human can still fail the result against a short rubric.

Worked example. Ticket: "Rewrite the onboarding README so a new hire can run the API locally without Slack."

# oracle-rubric.md
Timebox: 12 minutes.
Fail if any item is false:
1. `docker compose up` is documented with the exact service names in compose.yaml.
2. The port in the README matches the port in compose.yaml.
3. Secrets are referenced by environment variable names, not pasted values.
4. There is a verify section with one curl command and the expected status code.
5. No step depends on a screenshot in a chat thread.

Numbered workflow:

1. Write the rubric before the first prompt. If you cannot, this is not Leaf B.
2. Bound the model to the README and compose files only.
3. A reviewer applies the rubric once. Pass or fail. Not "looks better."
4. After a pass, extract the mechanical checks (port, service names) into a script so the next similar ticket can be Leaf A.

Leaf B is a bridge. If the same class of ticket repeats, the rubric should shrink into a command.

Leaf C — Sketch with wipe

When. You do not know the shape of the fix. You want the model to poke a disposable checkout. Nothing from the session is eligible for git am or cherry-pick until a later leaf applies on a clean tree.

Worked example. Question: "Is this timeout coming from the ORM, the proxy, or the client's retry budget?"

# sketch-session.sh — illustrative
set -euo pipefail
ROOT="$(mktemp -d /tmp/sketch.XXXXXX)"
trap 'rm -rf "$ROOT"' EXIT
git clone --depth 1 "$PWD" "$ROOT/repo"
cd "$ROOT/repo"
# agent may instrument logs here; residue dies with the trap
echo "sketch dir: $ROOT"

Numbered workflow:

1. State the wipe policy in the first line of the session notes.
2. Clone or worktree into a temp directory. Do not use your long-lived branch.
3. Allow the model to add prints, tighten log levels, or bisect.
4. Copy out notes, not patches, unless you re-enter Leaf A or B on a clean tree.

A separate server is useful here because the wipe boundary can be the machine or the worktree. Hope is not a boundary.

Leaf D — Do not start the loop

When. There is no failing command, no written rubric, and no wipe plan. Starting a loop here produces confident residue and a merge discussion you cannot score.

Worked example. Ticket: "Make the dashboard feel faster."

That sentence is not an oracle. Split it before any model runs.

• If you can measure p95 of /api/summary against a fixture, you have Leaf A.
• If you can fail a HAR capture against a 12-minute rubric, you have Leaf B.
• If you only want to poke the profiler, you have Leaf C.
• If none of those apply, the ticket is not ready. Write the measurement. Then come back.

Refusing the loop is a valid engineering output. It is cheaper than a plausible diff with no reject path.

A reusable gate

Keep a file the wrapper can read. Proposed layout:

# .agent-oracle.yaml
ticket: "CSV export BOM"
leaf: A
command: "python -m pytest -q tests/test_csv_export.py"
expect_initial: fail
allowlist:
  - app/export.py
  - tests/test_csv_export.py
budget:
  max_iterations: 8
  max_minutes: 20
wipe: false

# gate.py — illustrative; run before any agent process
from pathlib import Path
import subprocess, sys, yaml

cfg = yaml.safe_load(Path(".agent-oracle.yaml").read_text())
if cfg["leaf"] not in {"A", "B", "C"}:
    sys.exit("refuse: leaf D or unknown")

if cfg["leaf"] == "A":
    r = subprocess.run(cfg["command"], shell=True)
    if r.returncode == 0 and cfg.get("expect_initial") == "fail":
        sys.exit("refuse: oracle already green on HEAD")
    if r.returncode != 0 and cfg.get("expect_initial") == "fail":
        print("gate ok: oracle is red, loop may start")
        sys.exit(0)

if cfg["leaf"] == "B" and not Path("oracle-rubric.md").exists():
    sys.exit("refuse: rubric missing")

if cfg["leaf"] == "C" and not cfg.get("wipe"):
    sys.exit("refuse: sketch leaf requires wipe: true")

print("gate ok")

Run python gate.py as a pre-step. Non-zero exit means there is no first iteration.

This is the original artifact: a gate that fails closed. Models do not get a vote on whether the gate exists.

Decision table

Leaf	Oracle you must have	Allowed residue	Promotion
A Automated	Failing command on HEAD	Candidate patch in allowlist	PR + CI
B Rubric	Five-line fail list + timebox	Bounded files only	Human pass, then extract checks
C Sketch	Wipe command, no merge path	Temp clone or worktree	Notes only
D Refuse	None	None	Write the measurement first

If a row needs a cell you cannot fill, you are on the wrong leaf.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free model access and a free server option. Those two facts matter only after a leaf is chosen. They do not choose the leaf.

On Leaf A, a free model can iterate against the oracle command, and a free server can hold the checkout so a laptop is not the residue boundary. On Leaf C, the same server can be the wipe domain. On Leaf D, neither the model nor the server should start.

Do not treat free access as a reason to skip the gate. Cheap iterations without an oracle are how Friday's "all green" transcript happens. The YAML budget block above is local policy. It is not a product quota, a hardware spec, or a durability claim.

If you already pin oracles this way, MonkeyCode is one place you can run the bound loop. If your oracle and CI already live elsewhere, keep them.

Limitations

The tree does not estimate model quality. It does not rank vendors. It does not replace code review.

gate.py as shown does not sandbox the agent, does not compile allowlists into seccomp, and does not prove the oracle matches production. A green unit test can still miss a BOM on a second exporter path.

Flaky oracles will thrash Leaf A. If the command is not deterministic, fix the test or drop to Leaf B. Do not raise max_iterations to outvote a flake.

Who should not use this

Skip this approach if you cannot write a failing check or a five-line rubric, and you also cannot isolate a wipeable checkout. Skip it for an incident hotfix already under a human commander with a known rollback. Skip it if the "oracle" would need production data you are not allowed to copy onto a shared or free server.

Do not use Leaf C on a checkout that contains secrets, customer exports, or credential files. A trap wipe is not a data-loss prevention system.

Close

Name the oracle. Then name the leaf. Then, and only then, spend the first token.

The loop is a search. The oracle is the objective function. Without it you are scoring the model's tone.