Name an Eval Owner: A Wiki SOP When AI Writes the Tests Too

You have already seen this pattern on a shared lab board more than once this quarter. A teammate pastes a failing ticket into a coding assistant, accepts a full patch, and watches generated tests turn green. Nobody notices that the same session invented both the implementation and the assertions that bless it. The next on-call person inherits a merge that looks locally green and then fails in production.

This playbook names one human as Eval Owner before any AI-heavy pull request can merge. You keep the assistant, including free model access on a shared host, but you stop treating model-written tests as independent evidence. The artifact below is a one-page wiki SOP, a four-role handoff, and a small Python check you can run without the original chat.

The failure you are actually debugging

Shared free coding loops make this failure cheap to create and later expensive to notice during review. Someone drives a long session on a borrowed machine, dumps a patch plus tests, and leaves before standup. Reviewers skim the diff, trust the green squares, and never ask who authored the oracle that blessed the change. When the oracle and the code share one prompt, you have a tautology rather than a real test.

You do not need a new framework to interrupt that loop on a busy shared board. You need a named owner, a written bar, and a handoff that survives the person who closed the laptop. Write those three things on one wiki page so the next reader can replay the check without hunting through chat logs.

Current discussion about assistants drafting entire diffs makes this independence gap louder rather than smaller for shared labs. Faster patches increase the chance that tests and code are born together inside a single unreviewed thread. Your job is not to ban the assistant; your job is to keep the oracle out of that generating thread.

Four roles you paste into the wiki

Assign these names on the ticket before the first assistant prompt, not after the pull request already exists. If you skip the naming step, the Implementer silently becomes the Eval Owner and the Reviewer inherits a circular suite. Put the four handles where a tired reviewer will see them without opening extra documents or old threads.

1. Eval Owner. This person writes or selects checks that the coding session is not allowed to edit after freeze. They own fixtures, expected errors, and the replay command that a newcomer must be able to run.
2. Implementer. This person may use an assistant for code, refactors, and throwaway sketches against the frozen bar. They must not rewrite expected outputs so a convenient patch can turn the suite green.
3. Reviewer. This person confirms the suite is independent, readable, and actually fails against a known-bad stub. They run the wiki command on a clean checkout, not on the Implementer's already-tuned laptop.
4. Rollback Contact. This person stays reachable until the change is parked, reverted, or transferred to a named successor. Silence from this role means the pull request stays parked instead of drifting into an unowned merge.

If one person holds two roles, write that exception on the wiki page before work starts. Dual-hat work is allowed for tiny chores, but Eval Owner and Implementer should stay split for money, auth, or retained user data. A missing backup name is a parked ticket, not an invitation for whoever is awake to rewrite fixtures.

Decision table: is the eval independent?

Use this table in the pull request body so reviewers can score independence without rereading the whole assistant thread. If you cannot fill a row honestly, you do not yet have an eval, and the merge bar stays closed. The table is a conversation tool, not a score that you game by renaming generated tests as fixtures.

Question	Pass	Fail	Owner action
Who authored the fixtures?	A human Eval Owner, or a frozen file from before the session	The same chat that wrote the code	Reject and regenerate fixtures offline
Does a known-bad stub fail?	At least one mutation fails the suite	Suite stays green against an empty parser	Add a negative case before review
Can a newcomer replay without the chat?	A wiki command produces the same pass or fail	Replay needs the original thread or a hidden prompt	Record the command and pin the fixture hash
May the Implementer edit evals?	No, except typo fixes the Reviewer accepts	Yes, freely, in the same commit as the feature	Split commits or block merge

Numbered run: one page you can paste today

Copy the steps into your team wiki and fill the names before anyone opens a coding assistant. Keep the page to one screen so tired people will still follow it during a late review. If the page grows past a scroll, you have started writing a process novel instead of a runbook.

1. Name the Eval Owner on the ticket. Write the person's handle, timezone, and a backup before anyone opens an assistant. Do not start prompting until this line exists on the wiki page for that ticket.
2. Freeze the oracle before the assistant sees the production code path. Put fixtures, example payloads, and expected errors in a directory the Implementer cannot treat as draft. Share that path in the ticket so review does not hunt through personal scratch files.
3. Record the replay command in the same page. One shell line should run the independent suite from a clean checkout. If it needs extra env vars, list them beside the command instead of burying them in chat.
4. Run a negative check that a stub must fail. Point the suite at an empty or mutated parser and confirm it fails loudly. If the stub still passes, your oracle is too weak to serve as a merge bar.
5. Allow the Implementer to use an assistant after freeze. They may generate code against the frozen bar and iterate on implementation details. They may not rewrite expected outputs to match a convenient patch during that session.
6. Reviewer executes the wiki command on a clean checkout. Green results from the Implementer's laptop are not evidence for merge. Replay from the recorded command is the only evidence the Reviewer should accept.
7. Rollback Contact stays named until the next written handoff. If the owner goes offline, park the pull request rather than silently inheriting the bar. Transfer requires a new Eval Owner handle on the same wiki page.

One-page wiki template

Paste this block, then replace the brackets with handles, hashes, and a replay command that actually runs. Do not grow it into a novel, and do not hide the rollback contact below a fold of extra commentary. A short page that people use will beat a perfect page that nobody opens during review.

# Eval bar — [ticket id] — [date]

- Eval Owner: [@name] (backup [@name])
- Implementer: [@name]
- Reviewer: [@name]
- Rollback Contact: [@name] until [date/time]

## Frozen oracle
- Fixture path: `evals/invoice_totals.v1.json`
- Fixture sha256: [hash]
- Replay: `python evals/check_invoice_totals.py --fixtures evals/invoice_totals.v1.json`
- Negative replay: `python evals/check_invoice_totals.py --fixtures evals/invoice_totals.v1.json --mutate empty --expect-fail`

## Session notes
- Assistant host: [shared lab / local / other]
- System prompt frozen? yes/no
- Implementer edited evals? yes/no — if yes, Reviewer sign-off: [@name]

## Handoff
- Park / merge / revert: [choice]
- Next owner if parked: [@name]

A replayable check the chat does not own

The script below is an example you can run locally against a frozen oracle the chat does not own. It treats fixtures as the source of truth and refuses to proceed when the oracle file is missing. Label it as a template if your domain objects, locales, or error types differ from this invoice sketch.

# evals/check_invoice_totals.py
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path


def load_oracle(path: Path) -> list[dict]:
    if not path.is_file():
        raise SystemExit(f"oracle missing: {path}")
    payload = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(payload, list) or not payload:
        raise SystemExit("oracle must be a non-empty list")
    return payload


def parse_total(raw: str) -> str:
    """Replace this stub with the implementation under test."""
    digits = [ch for ch in raw if ch.isdigit() or ch in ".-"]
    if not digits:
        raise ValueError("no amount found")
    return "".join(digits)


def parse_total_empty(raw: str) -> str:
    return ""


def main() -> int:
    parser = argparse.ArgumentParser(description="Independent invoice total oracle")
    parser.add_argument("--fixtures", required=True)
    parser.add_argument("--mutate", choices=["none", "empty"], default="none")
    parser.add_argument("--expect-fail", action="store_true")
    args = parser.parse_args()

    fn = parse_total_empty if args.mutate == "empty" else parse_total
    cases = load_oracle(Path(args.fixtures))
    failures = []
    for case in cases:
        sample = case["input"]
        expected = case["expected"]
        try:
            got = fn(sample)
        except Exception as exc:  # example path; tighten in real code
            got = f"error:{exc.__class__.__name__}"
        if got != expected:
            failures.append((sample, expected, got))

    if args.expect_fail:
        if failures:
            print("negative check passed: oracle rejected the stub")
            return 0
        print("negative check failed: stub unexpectedly matched oracle")
        return 1

    if failures:
        for sample, expected, got in failures:
            print(f"oracle mismatch input={sample!r} expected={expected!r} got={got!r}")
        return 1

    print(f"oracle ok cases={len(cases)}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A tiny fixture file keeps the session honest because the assistant did not author it during the coding loop. Commit that file in a separate change, or at least before the Implementer starts prompting against the ticket. If the model proposes new expected values, the Eval Owner treats that proposal as a change request, not as truth.

[
  {"input": "Total due: USD 1,204.50", "expected": "1204.50"},
  {"input": "amount missing", "expected": "error:ValueError"}
]

Run the positive path and the negative path as two commands so the wiki replay line stays obvious to newcomers. Keep both commands on the wiki page rather than inside a personal alias that nobody else can find.

python evals/check_invoice_totals.py --fixtures evals/invoice_totals.v1.json
python evals/check_invoice_totals.py --fixtures evals/invoice_totals.v1.json --mutate empty --expect-fail

Hash the fixture after the Eval Owner freezes it, then paste the digest into the wiki page next to the path. If the Implementer's branch changes that hash, the Reviewer treats the eval as contaminated and restarts from freeze. Do not accept a verbal claim that the file only changed whitespace; the digest is the handoff artifact.

shasum -a 256 evals/invoice_totals.v1.json

Where a free shared coding host fits

Teams often run these loops on whatever machine is idle, then lose the thread that produced the patch. A shared coding environment with free model access and a free server option can host the Implementer's session without mixing personal billing. The Eval Owner still keeps oracles in version control that the chat cannot quietly edit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is one such option when you want the Implementer on a free model path. Keep fixtures in a repository the assistant does not control, even when the coding session runs on that shared host. If you try the SOP there, paste the wiki page first and keep the four handles visible in the pull request.

Limitations

This SOP does not prove the product is correct for every caller, locale, or production shape you will meet. It only proves the merge bar is independent of the generating session that proposed the patch. Property tests, contract tests, and production monitors still matter, and none of them replace a Reviewer who can read the domain.

Free model access and a free server option can change, pause, or move without notice from any vendor. You must not pin incident response to a particular host, queue, or assistant thread that might vanish overnight. Do not store secrets in the assistant thread, and do not treat generated tests as compliance evidence for auditors. The Python snippet is an example, not a library, and you should adapt types, errors, and locale rules before money paths.

Who should not use this approach

Skip this playbook if you already require independently authored tests and your review checklist already fails tautological suites. Skip it if a single volunteer is pretending to be four roles on a regulated workload with real customers. Skip it if your fixtures themselves come from production data you cannot legally copy onto a shared host.

You should also skip it when the change is a comment-only edit or a docs typo that cannot break behavior. Naming an Eval Owner for those chores trains people to ignore the page the next time a parser changes. Save the ritual for patches where a green suite could hide a shared wrong assumption about inputs.

Close the loop in standup, not in the chat

Ask one question at standup: who owns the eval bar for each open AI-heavy ticket on the board. If the answer is the model or whoever prompted last, you do not have a bar yet. Write the name, freeze the oracle, and only then let the assistant write code against a check it does not own.