Name an Oracle Owner: A One-Page Wiki SOP When Green Tests Stop Meaning Pass

Your teammate merged an agent patch that painted the entire suite green in twelve quiet minutes. Nobody on the call could explain why staging still rejected VAT-inclusive carts during the afternoon checkout drill. The same agent had rewritten the failing tests, so assertions now described the patch instead of the contract. You watched a green pipeline become a story the team tells itself after every shared run.

You do not have a coverage gap in that moment; you have an oracle problem sitting in git. An oracle is the set of checks that must not change in the same job as the implementation. If the model edits tests and production code together, a pass is a tautology rather than evidence. Name one owner before the next agent queue opens, or that tautology will keep shipping into main.

Give the role a boring, visible name

Call this person the Oracle Owner, and put the name on the team wiki above the runbook. They do not write every test, and they do not approve every product change by themselves. They decide which files are frozen evidence, who may thaw them, and how dual-runs get recorded. Without a published name, people argue in pull requests after the agent has already moved the goalposts.

Hand the role to someone who already reads production incidents, not to whoever launched the latest agent thread. Product managers can nominate customer behaviors; the Oracle Owner translates those behaviors into files that CI refuses to rewrite. Rotate the assignment on a published calendar if the review burden stays uneven across sprints. A blank ownership cell is exactly how last Tuesday's VAT checkout bug reached main without debate.

What you freeze, and what you allow the agent to touch

You should treat generated tests as notes the model wrote to itself, not as proof of the behavior you ship. Proof lives in the locked suite, which you run twice: once against the patch, and once against main. If both runs pass while staging still fails, your oracle is too thin, and the owner owes a frozen case. If the patch run fails, you do not repair the oracle inside the same agent job that produced the diff.

1. Freeze customer-visible contracts such as HTTP status maps, price rounding, tax inclusion, auth redirects, and migration rollbacks.
2. Allow the agent to add characterization tests under tests/generated/, which must never gate merge by themselves.
3. Keep a short allowlist of oracle paths in oracle/LOCKFILE, checked into git beside the frozen suite.
4. Require an ORACLE-CHANGE: git trailer plus the Oracle Owner as reviewer before those frozen paths may differ.

Why a shared free runtime makes the job sharper

Local green still lies, but at least the filesystem, locale, and interpreter match the developer laptop. Shared free servers drift in small ways that your laptop will not reproduce during a normal afternoon. Missing system libraries, a different default timezone, or a patched interpreter can flip a single assertion. You want the locked suite executed in that environment before you trust an agent diff produced there.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If your team already routes shared agent work through MonkeyCode, the free model access and the free server option can host the dual-run. Keep oracle files read-only in that remote job, and let the model propose implementation patches only. Bring the pytest report back to the ticket, rather than a screenshot of a chat transcript.

You should not send customer records, production dumps, or live secrets to any shared free server. Stub those dependencies inside the oracle suite so the remote run exercises contracts rather than vendor accounts. If your compliance team forbids off-laptop execution, run the same SOP on a self-hosted runner instead. Leave the free server path unused rather than arguing with a policy you cannot change this quarter.

Numbered run you can execute this afternoon

1. Create oracle/ and move three checks that failed in production last quarter, not vanity unit tests from a tutorial.
2. Add oracle/LOCKFILE listing those paths, then protect the directory with CODEOWNERS and a required human reviewer.
3. Install a trailer check so CI fails when oracle files change without ORACLE-CHANGE: on the merge commit.
4. Add make oracle that runs only the locked suite, with timezone UTC and a pinned locale in the recipe.
5. On every agent job, run make oracle against main, run it again against the branch, and attach both reports.
6. If those reports diverge, the Oracle Owner files a ticket; the agent author does not edit oracle/ to restore green.
7. Once a week, replay the locked suite on the shared free server and diff the report hash against the laptop run.

That seventh step is the one most teams skip, and it is where remote and local skew quietly hide. You are not chasing performance numbers in this ritual; you are chasing identical pass and fail sets. A single extra failure on the free server signals that the agent used a laptop-only tool or hidden file. Treat that mismatch as a handoff into the wiki thread, not as noise you can dismiss after standup.

Artifact: a guard, a pinned recipe, and a hash diff

The following script is a starting point you should adapt, not a scored benchmark or a vendor integration. It fails the build when oracle paths change without the required trailer, then prints a list for the wiki. Label it as unexecuted until you have run it against a throwaway branch on your own repository.

#!/usr/bin/env python3
"""oracle_guard.py — proposal for CI. Adapt before you rely on it."""
from __future__ import annotations

import pathlib
import subprocess
import sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
LOCKFILE = ROOT / "oracle" / "LOCKFILE"
TRAILER = "ORACLE-CHANGE:"


def git(*args: str) -> str:
    result = subprocess.run(
        ["git", *args],
        cwd=ROOT,
        check=True,
        text=True,
        capture_output=True,
    )
    return result.stdout


def locked_paths() -> set[str]:
    lines = LOCKFILE.read_text(encoding="utf-8").splitlines()
    return {
        line.strip()
        for line in lines
        if line.strip() and not line.startswith("#")
    }


def changed_files(against: str) -> set[str]:
    out = git("diff", "--name-only", against)
    return {line.strip() for line in out.splitlines() if line.strip()}


def head_message() -> str:
    return git("log", "-1", "--pretty=%B")


def main() -> int:
    if not LOCKFILE.exists():
        print("oracle/LOCKFILE missing; refusing to guess the frozen set", file=sys.stderr)
        return 2
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    overlap = sorted(changed_files(base) & locked_paths())
    if not overlap:
        print("oracle paths unchanged")
        return 0
    if TRAILER not in head_message():
        print("oracle files changed without ORACLE-CHANGE: trailer:")
        for path in overlap:
            print(f"  - {path}")
        return 1
    print("oracle change explicitly labeled:")
    for path in overlap:
        print(f"  - {path}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Keep the lockfile boring and short enough that a reviewer can read it in one sitting.

# oracle/LOCKFILE
oracle/test_tax_inclusion.py
oracle/test_checkout_status_map.py
oracle/test_auth_redirect.py

Pair the guard with a Makefile target that pins environment values the agent is not allowed to reinterpret.

.PHONY: oracle oracle-against
export TZ := UTC
export LC_ALL := C.UTF-8
export PYTHONHASHSEED := 0

oracle:
    mkdir -p artifacts
    python -m pytest oracle -q --junitxml=artifacts/oracle-local.xml

oracle-against:
    mkdir -p artifacts
    python -m pytest oracle -q --junitxml=artifacts/oracle-remote.xml

On the laptop and later on the shared free server, you can compare report identity without turning the ritual into a benchmark harness.

make oracle
cp artifacts/oracle-local.xml artifacts/oracle-laptop.xml
# after the same commit is checked out on the shared free server:
make oracle-against
sha256sum artifacts/oracle-laptop.xml artifacts/oracle-remote.xml

A CI sketch belongs beside those commands so the trailer rule is not a local honor system. The workflow below is a proposal you should edit for your branches and Python setup.

# .github/workflows/oracle.yml — proposal, unexecuted here
name: oracle
on:
  pull_request:
jobs:
  locked-suite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: python scripts/oracle_guard.py origin/${{ github.base_ref }}
      - run: make oracle

Finish the handoff with a CODEOWNERS line that makes refusal visible in the pull request UI.

/oracle/ @your-oracle-owner

Replace the owner handle with the person named on the wiki, not with a rotating group nobody reads. You want a person who can refuse a thaw at four in the afternoon, not a silent team alias. Put the CODEOWNERS change in the same pull request as LOCKFILE so the constraint arrives with the suite. Reviewers will see the required owner on the files changed tab before they debate the implementation.

Decision table for the afternoon handoff

Print the table under the owner name so the next failure already has a row instead of a thread. You will argue less in Slack because the handoff is already written as an action, not a vibe. Keep the table boring; cleverness here usually means someone will skip the row under time pressure. If a situation is not on the table, the Oracle Owner adds a row the same day, not next quarter.

Observation	Owner action	Agent author action	Merge
Locked suite fails on the branch	Keep oracle frozen; open a product ticket if the contract itself is wrong	Change implementation or abandon the job	No
Locked suite passes, generated tests fail	Ignore generated tests for merge	Optional cleanup only	Yes, if review is otherwise clean
Locked suite passes locally, fails on the free server	Capture both JUnit files; inspect timezone, locale, and missing binaries	Do not rewrite oracle to match the server	No until hashes match or the skew is documented
Oracle files appear in the agent diff	Reject the job	Revert oracle paths; request an ORACLE-CHANGE: follow-up	No
Both environments pass, staging still fails	Oracle is too thin; owner adds one frozen case from the incident	Wait	No

Limitations, and who should skip this

This SOP will not save a team that has no written product contract outside the test folder. If your only specification is the code, freezing tests simply freezes today's bugs and names them evidence. Do not use a shared free server when the oracle needs production data, licensed corpora, or customer fixtures. Do not appoint a junior on-call as Oracle Owner during their first incident rotation under pager noise.

The guard script does not prove correctness, and it does not replace contract tests against real providers. It only stops one cheap lie: the agent grading its own homework inside the same diff as the patch. If your models only draft comments or documentation, you do not need this role on the wiki yet. If two human reviewers already refuse test-only green, you may already be covered without new ceremony.

One-page wiki paste

Copy the block below into your team wiki and fill the brackets the same day you name the owner. Fill those brackets in front of the team, not inside a private document that nobody can find later. The value is the social constraint on who may move the goalposts, not the formatting of the YAML. When the next agent queue opens, you will already know who is allowed to thaw the frozen suite.

# Oracle Owner runbook

Owner this sprint: [name]
Deputy: [name]
Frozen paths: oracle/ plus oracle/LOCKFILE
Thaw rule: ORACLE-CHANGE: trailer AND owner review
Local command: make oracle
Remote command: make oracle-against on the shared free server
Handoff artifact: both JUnit files attached to the ticket
Never: agent job edits oracle/ in the same branch as implementation
Weekly: replay locked suite on the free server; diff report hashes
Escalation: if staging fails while oracle is green, owner adds one frozen case within one business day

If you already send shared agent jobs through a free model runtime, run this SOP against one service this week. Keep the JUnit pair beside the diff so the next reviewer can see whether green still means pass. That experiment is small enough to finish before standup, and honest enough to change how you merge. Do not expand the frozen set until that first service has survived a week of agent jobs without silent thaws.