Verification Needs Its Own Failure Domain

Generated tests cannot vouch for generated code. They fail in the same places. You need a separate verification domain.

A model that writes a function also writes its exam. That is one author grading itself. You would not ship that in any other system.

Think of a locksmith who also prints the inspection sticker. The door looks certified from the hallway. The sticker never left the same workshop. Your AI pipeline often works exactly that way.

You already split generate from apply in other reviews. This piece is not that split. This is about the oracle that claims the patch is safe.

Constraints you actually have

You do not control the model's private checks. You control what may enter main. That is the only honest constraint you own.

The context window is not a specification. It is mixed, stale, and incomplete. Treat it as contaminated input, not as law.

Time fights you as well. A generated suite can pass tonight. It can miss the bug you will hit on Friday.

Cost fights you too. Cheap tokens invite extra tests from the same session. Extra tests from the same author add theater, not coverage.

You also cannot prove a hidden property the corpus never stated. Silence is not green. Green requires an oracle you already trusted.

Data flow that stays honest

Keep three streams, and never braid them. The corpus is frozen and human-owned. The candidate is generated and untrusted. The verdict is computed on a third host.

You send the model a task envelope only. You receive a patch, nothing else. You refuse tests, scripts, and CI edits from that reply.

The verifier host clones a clean tree. It applies the patch in a sandbox. It runs only the frozen corpus against that tree.

If the corpus is silent on a change, you fail closed. New behavior without an old oracle is a hole. You write the oracle first, then you generate.

Here is a proposed local gate. Label it unexecuted until you wire it. Do not treat it as a certified toolchain.

#!/usr/bin/env bash
# verify_gate.sh — proposed local gate, not production gospel
set -euo pipefail

PATCH=${1:?patch file required}
CORPUS=${2:?frozen corpus directory required}
ROOT=$(git rev-parse --show-toplevel)
WORKDIR=$(mktemp -d)
trap 'rm -rf "$WORKDIR"' EXIT

git clone --local "$ROOT" "$WORKDIR/tree"
git -C "$WORKDIR/tree" apply --check "$PATCH"
git -C "$WORKDIR/tree" apply "$PATCH"

# Product code may change. The scorekeeper may not.
if awk '/^diff --git /{f=0} /^diff --git .*\/(tests|spec)\// {f=1} f && /^\+\+\+ /{print}' "$PATCH" | grep -q .
then
  echo "refuse: patch touches tests; corpus stays frozen" >&2
  exit 2
fi

cd "$WORKDIR/tree"
python -m pytest "$CORPUS" -q --maxfail=1

That script encodes one architectural rule. The patch may change product code. It may not smuggle its own scorekeeper into the tree.

Pair the gate with a contract the corpus already owns. The generator should never receive this file if you can help it. Hidden oracles beat prompt-fed oracles.

# contracts/test_quote_shape.py
# Proposed example. Wire it before you invite a generator.

from app.billing import quote


def test_quote_never_goes_negative():
    total = quote(items=[], coupon="WELCOME")
    assert total >= 0


def test_quote_rejects_unknown_sku():
    try:
        quote(items=[{"sku": "nope", "qty": 1}], coupon=None)
    except ValueError:
        return
    raise AssertionError("unknown sku must fail")

You own those checks before any model runs. They describe money, not style. A generator that never saw them cannot teach to the test.

Move the patch across the boundary as a file. Do not pipe a chat transcript into pytest. Files have hashes. Chat logs do not.

# proposed flow on your verifier host
sha256sum candidate.patch > candidate.patch.sha256
./verify_gate.sh ./candidate.patch ./contracts

If the hash and the verdict disagree later, you have a story. If you only kept the chat, you have fog.

Failure domains, drawn as rooms

Picture three rooms with locked doors. Room A writes candidates. Room B stores the corpus. Room C applies merges.

Room A can burn down. You lose drafts and prompts. That loss is acceptable. Drafts are cheap. Oracles are not.

Room B must not share a filesystem with A. Shared disks are shared fate. Shared prompt history is shared fate too.

Room C accepts only artifacts that B already blessed. It does not chat with a model. It does not fetch "just one more test."

Most teams collapse A and B onto one laptop. That laptop then talks to CI with your identity. One prompt injection now poisons code and proof together.

This is not a review of the generator host itself. You already know that host is a blast radius. This review is about the oracle living next door and pretending it is independent.

If verification shares memory, credentials, or logs with generation, it is not independent. A unit test written in the same session is a roommate, not a referee. Roommates do not audit the lease.

Network policy should make the rooms boring. Room A reaches a model endpoint only. Room B reaches nothing outbound. Room C reaches your registry, not a chatbot.

Logs need the same split. Generator traces stay in A. Corpus hashes stay in B. Merge tickets in C cite both, and never mix the files.

# proposed record, written by the verifier, not the model
patch_sha256:  9f3c…
corpus_sha256: 41aa…
pytest_exit:   0
verifier_host: verify-07
generator_host: (cited, never trusted)

That record is the architecture. The model is a printer in Room A. Printers do not sign the building permit.

Where a disposable generator host fits

You still need somewhere to produce candidates. That somewhere should be disposable. It should never hold the frozen corpus.

MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Park generation there so Room A is not your laptop.

Keep the oracle off that box. Do not upload contracts to a throwaway server. Do not let the apply identity live there either.

A short pattern looks like this. Generate on the remote box. Pull a patch file home. Verify on a host that never held the prompt.

# proposed; your generator command stays yours
ssh generator 'python gen_candidate.py > /tmp/candidate.patch'
scp generator:/tmp/candidate.patch ./candidate.patch
./verify_gate.sh ./candidate.patch ./contracts

If the free server vanishes, you lost drafts. You did not lose the exam. That is the point of the room split.

Do not send production snippets to any shared generator to "make tests." That inverts the flow. The exam would then be written from the crime scene.

What you should change next

Sign the corpus and verify the signature in CI. A writable tests directory is an unsigned exam. Unsigned exams invite the author back into the grading room.

Add a path denylist in the merge queue, not only in a laptop script. The script above is a teacher. The queue is the lock.

# proposed merge-queue fragment
policy:
  refuse_if_patch_touches:
    - contracts/**
    - tests/**
    - .github/workflows/**
  require:
    - contracts_signature
    - verify_gate_exit_zero

Then grow oracles that are not example-based. Property tests and golden files age better than one happy-path pytest. Generated unit tests age like unused coupons.

If you can afford it, run two verifiers. Same patch, two hosts, one corpus hash. Disagreement is a signal. Agreement is still not proof of missing properties.

Mutation testing against the frozen corpus is the next honest metric. If killing a line does not fail a contract, you lacked an oracle. You did not lack another generated test.

Limitations, and who should not bother

This split fails when you have no frozen corpus. A greenfield spike still needs a human oracle first. Do not generate the product and the proof in one sitting and call it architecture.

This also fails when the product is the test suite. Fuzz harnesses and compiler tests need a different shape. Do not copy this gate onto those trees without redesign.

Regulated or safety-critical work needs more than pytest and a shell script. This article does not offer a certified method. Do not file it as one.

Solo prototypes can ignore the three rooms for a day. Do not ignore them the day you attach CI credentials. Theater starts at the first green badge you did not earn.

You should not use a remote generator for private data you cannot expose. Free compute is not a data policy. Room A still egresses whatever you paste.

Verification is a failure domain, not a plugin. If the author still writes the exam, you are grading a mirror. Keep the mirror in Room A, and keep the exam in Room B.