A free model pass is a draft signal, not a release gate. The golden fixture, its content hash, and the bit that says a change may ship stay on infrastructure the team already owns and can replay without a vendor round trip. A hosted assistant can suggest a patch, and a spare host can run a throwaway smoke. Neither is the system of record for whether the suite passed.
The blur starts when the assistant feels faster than review. A flaky assertion fails, the model rewrites the expected file so the assertion goes quiet, and a scratch host prints a green line that someone pastes into the release note. The lab notebook left the building, and the wax seal left with it. Regression evidence only holds when the same bytes can be hashed tomorrow on a machine the team administers.
That property belongs to storage and process, not to how confident the draft sounded. A helpful explanation of a stack trace does not become an oracle because it arrived quickly. The oracle is the frozen expected bytes plus a scorer the team can run offline. Speed of the sketch is a separate concern, and mixing the two is how a green log replaces a test.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access is a reasonable place to ask for a patch sketch or a failure explanation. A free server option is a reasonable place to boot a process the team can afford to throw away if the run fails. Those roles stop at the draft, and they do not extend to custody of fixtures, to the scorer, or to the pass bit other jobs trust.
Red flags, not outages
Red flags show up as ownership slips rather than as dramatic outages. A fixture appears in the assistant transcript and never lands in the repository as a reviewed diff, or the only copy of expected output sits on a host the team does not administer. The pass bit is a sentence the model wrote about its own patch, or a log line from a runner that prints green without recomputing hashes. Release timing starts to depend on whether that path answers during the window the team actually has.
Any one of those slips is enough to pull the gate back. Together they mean the suite is no longer a test, and a rumor with a timestamp will not survive the next incident review. The fix is not a stricter prompt and not a longer system message. Careful wording does not restore a file that was never committed, and it does not turn an unadministered disk into a system of record.
Three homes for the pass bit are easy to confuse because all three can print the word passed. The model transcript is cheap to read and useless as authority, since a later reader cannot replay the bytes that were judged. Disk on a free server is a fair place for a crash dump and a bad place for the only oracle, because retention and rebuild of that disk sit outside the release job. The primary runner that already produces release artifacts is slower to sketch on, and it is the only home that can refuse a drifted fixture without asking a draft for permission.
The anti-pattern is letting the transcript or the free host write a file the primary is expected to trust. Later jobs then stop opening fixtures and start reading a boolean. Booleans travel well, and they hide which tree was scored, which is why the owned pass has to carry the hash rather than a color.
A seal the draft cannot refresh
A better arrangement is dull on purpose. Fixtures live in the repo beside the code that consumes them, and a local command hashes those files before any assistant proposes an edit. The hash ignores mtime, file owner, and directory walk order, so two checkouts of the same bytes agree. The assistant may propose a source change or a fixture change, but the proposal is a diff, not a commit, and it has no path that updates the seal.
A free server may run the candidate in a scratch directory whose outputs are labeled untrusted. The owned runner checks the hash, runs the scorer the team already maintains, and is the only writer of the pass file. If the free path is down, releases slow by the time a human needs to sketch a patch. They do not stop because the oracle went missing, which is the failure this split is built to prevent.
The Python below is a proposal, not a captured run. It has not been executed against a live free server, and it states no model name, quota, or timing result. It refuses an untrusted smoke result as authority, and it stops if the fixture tree drifts after the seal. NUL separators keep a filename from colliding with contents, and sorting makes the digest independent of filesystem order.
#!/usr/bin/env python3
"""Proposal: seal fixtures locally, then reject an untrusted pass bit.
Not executed against a live free server. Point ROOT at a temp checkout.
primary-score.json must be written by the owned runner after its own suite.
A file that merely claims runner=primary-runner is not identity proof.
"""
from __future__ import annotations
import hashlib
import json
import sys
from pathlib import Path
ROOT = Path("fixtures")
SEAL = Path("fixtures.seal.json")
UNTRUSTED = Path("smoke-result.json")
PRIMARY_SCORE = Path("primary-score.json")
OWNED_PASS = Path("owned-pass.json")
OWNER = "primary-runner"
def read_json(path: Path) -> dict:
try:
data = json.loads(path.read_text())
except json.JSONDecodeError as exc:
raise SystemExit(f"bad json in {path}: {exc}") from exc
if not isinstance(data, dict):
raise SystemExit(f"expected object in {path}")
return data
def tree_hash(root: Path) -> str:
digest = hashlib.sha256()
if not root.is_dir():
raise SystemExit(f"missing fixture dir: {root}")
files = sorted(p for p in root.rglob("*") if p.is_file())
if not files:
raise SystemExit("empty fixture dir; refuse to seal a vacuum")
for path in files:
rel = path.relative_to(root).as_posix()
digest.update(rel.encode())
digest.update(b"\0")
digest.update(path.read_bytes())
digest.update(b"\0")
return digest.hexdigest()
def seal() -> None:
payload = {"algo": "sha256", "fixture_hash": tree_hash(ROOT), "owner": OWNER}
SEAL.write_text(json.dumps(payload, indent=2) + "\n")
print(f"sealed {payload['fixture_hash']}")
def gate() -> None:
if not SEAL.is_file():
raise SystemExit("no seal; refuse to score")
sealed = read_json(SEAL)
current = tree_hash(ROOT)
if current != sealed.get("fixture_hash"):
raise SystemExit("fixture drift after seal; draft cannot refresh the oracle")
if sealed.get("owner") != OWNER:
raise SystemExit("seal owner is not the primary runner")
if UNTRUSTED.is_file():
smoke = read_json(UNTRUSTED)
if smoke.get("runner") != OWNER:
print("ignored untrusted smoke; it cannot set the pass bit")
if not PRIMARY_SCORE.is_file():
raise SystemExit("primary has not scored; refuse to promote a draft")
score = read_json(PRIMARY_SCORE)
if score.get("runner") != OWNER or score.get("passed") is not True:
raise SystemExit("primary score missing or failed")
if score.get("fixture_hash") != current:
raise SystemExit("primary score hashed a different tree")
OWNED_PASS.write_text(json.dumps({
"passed": True,
"runner": OWNER,
"fixture_hash": current,
}, indent=2) + "\n")
print("owned pass written")
if __name__ == "__main__":
cmd = sys.argv[1] if len(sys.argv) > 1 else "gate"
{"seal": seal, "gate": gate}[cmd]()
A scratch directory makes the boundary visible before anyone wires a host. The first gate is supposed to fail closed, and the gate after the fixture edit is supposed to fail closed as well. Those failures are the control, not bugs in the demo.
mkdir -p fixtures
printf 'expected: 2\n' > fixtures/add.out
python3 seal_fixtures.py seal
printf '%s\n' '{"passed":true,"runner":"free-smoke"}' > smoke-result.json
python3 seal_fixtures.py gate || echo "blocked_without_primary_score=$?"
python3 - <<'PY'
import json
from pathlib import Path
seal = json.loads(Path("fixtures.seal.json").read_text())
Path("primary-score.json").write_text(json.dumps({
"passed": True,
"runner": "primary-runner",
"fixture_hash": seal["fixture_hash"],
}) + "\n")
PY
python3 seal_fixtures.py gate
printf 'expected: 3\n' > fixtures/add.out
python3 seal_fixtures.py gate || echo "blocked_after_drift=$?"
The transcript below is illustrative. It is not a log captured from a run, and the hex digest will differ if the fixture bytes differ.
sealed <sha256 of fixtures/>
ignored untrusted smoke; it cannot set the pass bit
primary has not scored; refuse to promote a draft
blocked_without_primary_score=1
ignored untrusted smoke; it cannot set the pass bit
owned pass written
fixture drift after seal; draft cannot refresh the oracle
blocked_after_drift=1
Read the owned pass after the successful gate and the free-smoke identity is absent. The file names the primary runner and repeats the sealed hash, which is the only identity later jobs should accept. Drift the oracle the way an eager rewrite would, and the next gate stops before it can refresh the seal. A human commits the fixture change, a reviewer reads the diff, and only then does someone run seal again on the primary.
The primary-score.json written in that demo is a stand-in so the gate can be exercised. A real job replaces that file with the output of the suite the team already trusts, and the stand-in must never be copied into CI. The free server JSON can sit in the build log as a hint about what broke. Nothing in gate ORs that passed field into the result, and the omission is deliberate, because a hint that can flip a boolean will be used as a boolean the first time the primary is slow.
The scratch host receives a copy of the candidate, not a write credential for the seal, the primary score, or the owned pass. If those paths are mounted into the free server, the split is theater. A forged primary-score.json with the right runner name is just another untrusted green line. The gate cannot detect a lie about identity that the filesystem itself allowed.
Wire the same rule into the job that already publishes artifacts. The shell is illustrative, not a workflow from a specific vendor, and it assumes the proposal lives at seal_fixtures.py beside a real primary score.
set -euo pipefail
test -f fixtures.seal.json
test -f primary-score.json
python3 seal_fixtures.py gate
python3 - <<'PY'
import json
from pathlib import Path
seal = json.loads(Path("fixtures.seal.json").read_text())
owned = json.loads(Path("owned-pass.json").read_text())
assert owned["runner"] == "primary-runner"
assert owned["passed"] is True
assert owned["fixture_hash"] == seal["fixture_hash"]
print(owned["fixture_hash"])
PY
If gate exits non-zero, the job stops. There is no fallback that asks a free model whether the red result was probably fine, because that fallback is the original bug in a softer voice. A sketch can still happen earlier, on a branch, where a wrong suggestion costs a review comment instead of a shipped oracle.
When to walk away
Exit is a set of conditions, not a mood. Take the free path out of the gate when a fixture hash changes without a human commit, when the smoke host holds the only expected output, or when two runs of the same candidate disagree and nobody can name the bytes that were scored. Also take it out when the release window is shorter than the worst stall the team has actually observed on that path. An unmeasured hope is not an SLO, and this note does not invent one.
The same exit applies when the suite touches secrets, live payments, production rows, or prompts that still contain customer text. Those bytes should not move to a host the team does not administer in exchange for a faster comment. The alternative is the owned runner alone, with the assistant kept as an editor. That editor never sees the pass bit and never receives a credential that can rewrite the seal.
This approach is a poor fit for several teams, and a halfway adoption is worse than a skip. A group with no primary runner should not rename a free server into one, because the missing piece is custody rather than a hostname. A group whose fixtures hold customer data, tokens, or unredacted prompts should not upload that tree for a faster sketch. A group that cannot tolerate a manual patch when the free path stalls should keep the assistant off the release path.
This article does not state a quota, a hardware shape, a duration, or a promise that a free option remains. Those details change, and a post is the wrong place to freeze them. Hash equality does not prove domain correctness either. It only proves the oracle did not move under the draft, so the real scorer still belongs on the primary, in the language CI already runs, with failures a human can open.
Used inside those limits, MonkeyCode's free model access is the sketch step: a candidate diff, read by a person, and dropped on a branch the primary will score. The free server option is the scratch smoke, a place to run the candidate where a crash does not delete the seal. For a team that already owns CI, the next step is small. Point the sketch at that free model access, keep the seal file on the primary, and treat every other green line as unread until the owned gate reprints it.
Top comments (1)
tr.ee/dev-to