When a Free Lane Must Not Sign the Eval Report

A Friday deploy waits on a single eval report. The patch came from a free model lane. That same lane then scored the patch as safe.

No second examiner signed the written ship result. The pattern looks efficient to a tired reviewer. It is not a valid release gate today.

A grader that shares quota and host cannot disprove its own work. Shared prompt history makes a clean verdict unlikely.

The failure this guide targets

Agent drafts now land faster than review staffing grows. A green self-score becomes the only stored evidence. The free lane then holds both the draft and the verdict.

This guide is a written field refusal list. It names red flags, safer placements, and exit criteria. It does not claim a measured model ranking.

Why a self-score fails closed

Shared identity

An examiner must be able to fail the author. That power disappears when both roles share one key. A quota error then threatens the verdict itself.

Disposable storage

A free server adds a second operational trap. Scratch disks get wiped, resized, or quietly reclaimed. The only trace of a failure can vanish with them.

Inflated scores

Perfect scores deserve extra doubt rather than extra trust. Case leakage, skipped rows, and hint-filled prompts inflate them. A frozen content hash is the minimum countermeasure here.

Red flags

Block auto-apply when any flag below is true. Treat the list as a hard stop, not a suggestion.

• The scorer and the author share one API key.
• The eval harness runs only on the free server.
• A zero-mistake score has no frozen case set.
• The same job can delete failures after scoring.
• The report is the only artifact kept before apply.
• Prompt text quietly includes the expected answers.
• Quota pressure can skip cases to finish the run.
• A human cannot replay the case set offline.
• The grader principal is a rename of the author.
• Security cases are marked optional near the quota.

One flag is enough to block auto-apply. Two flags mean the lane is only an author. Do not promote that lane into the examiner role.

Better alternatives

Keep generation on the cheap lane when blast radius stays small. Move the oracle to another principal and another host. Keep raw traces where the free server cannot delete them.

1. Freeze the case set in git with a content hash.
2. Run the scorer under a different network path.
3. Require a human signature for auth, data, or money diffs.
4. Store traces in an append-only bucket off the free host.
5. Treat a perfect score as a sampling prompt, not a pass.
6. Fail the job when any security case is skipped.
7. Record the grader key id beside the case-set hash.

A free model may still draft the patch. A free server may still host a scratch replay. Neither may become the system of record for the verdict.

Decision table

Use this table before any production apply step. The middle column describes a disposable sandbox only. The right column describes production authority you must withhold.

Signal	Free lane may	Free lane must not
Draft a unit-test sketch	Yes, as a marked proposal	Merge it unmarked
Score its own patch	No	Sign the ship report
Host a scratch replay	Yes, if the disk is disposable	Store the only trace
Skip cases near quota	No	Drop security cases first
Name a human reviewer	Yes, as plain text	Impersonate that reviewer
Hold release evidence	No	Be the only archive

A gate you can run

The snippet below is a labeled proposal only. It was not executed against a live account for this draft. It checks identity, case-set hash, and storage separation.

# Proposal only. Not an executed production test.
import hashlib
import json
from pathlib import Path

FREE_LANE_PRINCIPALS = {"free-model", "free-server"}

def case_set_hash(path: Path) -> str:
    payload = path.read_bytes()
    return hashlib.sha256(payload).hexdigest()

def allow_apply(report: dict, expected_hash: str) -> tuple[bool, str]:
    author = report.get("author_principal", "")
    grader = report.get("grader_principal", "")
    if not author or not grader:
        return False, "missing principal"
    if author in FREE_LANE_PRINCIPALS and grader == author:
        return False, "self-grade on a free lane"
    if author in FREE_LANE_PRINCIPALS and grader in FREE_LANE_PRINCIPALS:
        return False, "both roles still sit on the free lane"
    if report.get("case_set_sha256") != expected_hash:
        return False, "case set hash mismatch"
    if report.get("trace_store") in FREE_LANE_PRINCIPALS:
        return False, "only copy of traces is on the free lane"
    skipped = report.get("skipped_security_cases")
    if skipped is None or skipped > 0:
        return False, "security cases were skipped or omitted"
    if report.get("blast") == "high" and not report.get("human_signed"):
        return False, "high blast change lacks a human signature"
    return True, "oracle separated from the free lane"

if __name__ == "__main__":
    expected = case_set_hash(Path("cases/frozen.jsonl"))
    sample = {
        "author_principal": "free-model",
        "grader_principal": "free-model",
        "case_set_sha256": expected,
        "trace_store": "audit-bucket",
        "skipped_security_cases": 0,
        "human_signed": False,
        "blast": "high",
    }
    ok, reason = allow_apply(sample, expected)
    print(json.dumps({"ok": ok, "reason": reason}))

Pin the frozen case file before you run the script. A false result must fail the pipeline run. Do not catch that error and continue the apply.

sha256sum cases/frozen.jsonl
python gate.py
# Proposal expectation, not a captured live log:
# {"ok": false, "reason": "self-grade on a free lane"}

Fail closed in CI

Wire the script as a required check, not a comment. A proposal step looks like the block below. It is not a workflow taken from a live account.

# Proposal only. Not a published workflow.
- name: reject self-graded free lane
  run: python gate.py

Two free principals still fail the second guard. A rename must be added to the set before the next run. Otherwise the denylist becomes a mere label game.

Fixture checklist

Walk this list on every pipeline change you ship. Mark a failed row as failed closed, not skipped.

• Self-grade with a matching hash returns false.
• A renamed free principal must still return false.
• Hash mismatch returns false even with a new grader.
• A free-server trace store must always return false.
• A skipped security case must always return false.
• High blast without a human signature returns false.
• External grader, external traces, and a signature return true.
• Missing report keys return false, not a default pass.

Missing keys need an explicit guard in the runner. The sample function already fails some empty gaps. Extend that function rather than wrapping a bare except.

Where a workshop lane still fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator notes for this draft describe MonkeyCode free model access. Those same notes also describe a free server option.

The notes state a free pool of ten million tokens. Confirm that figure and any expiry in current docs. This draft does not treat the number as a benchmark.

It also does not treat the pool as a permanent quota. Use that access to draft patches and sketch new cases. Use the free server for a scratch replay you can delete.

Keep the signed verdict and the trace bucket elsewhere. A workshop lane is still useful after the split. It is not a quiet second production region.

If current docs show a smaller pool, drop those steps. If docs show no free server, drop the scratch host too. Do not hard-code an allowance the vendor can change.

Exit criteria

Move the examiner role off the free lane when any line is true. Leave drafting in place only when the oracle sits elsewhere.

• The case set now includes customer data or live secrets.
• Apply rights can change auth, billing, or data retention.
• The free host cannot prove trace durability for audits.
• Quota errors start skipping cases instead of failing.
• Two releases passed only because of a self-score.
• Reviewers cannot replay a failure without the free host.
• The same key can both write the score and delete it.
• A renamed free principal bypassed the denylist once.

Exit means you split the principals and the storage. It does not always mean you abandon cheap drafting. Re-run the fixture list after every such split.

Who should not use this split

Skip this pattern if you have no second principal. A paper grader name on the same key is theater. The script will only echo the lie you typed.

Skip it for homework with no apply step. A local notebook does not need this gate. Spend that time on clearer cases instead of ceremony.

Skip it when policy already forbids model-written code. This guide assumes a draft lane already exists. If drafts are banned, the middle column is already closed.

Do not park production on the free server to dodge cost. That move recreates one failure domain under a new name. Cost relief is not an audit design by itself.

Limitations

The script checks strings you pass into it. It cannot discover a renamed free principal alone. Map those names from your real identity provider.

A content hash does not prove case quality. A frozen weak set still matches its own hash. Review the cases on a schedule you can defend.

No latency, cost, or accuracy numbers are claimed here. None of those figures were measured for this draft. Do not cite this page as a product benchmark.

Token allowances change without much notice in many products. A stated pool of ten million tokens can be revised. Re-read the vendor docs on the day you ship.

A second model on the same free pool shares one failure domain. Prefer a different host plus a human for high-blast changes. Model diversity alone is not real independence here.

This guide does not cover legal holds or webhook signatures. Lock leases and cleanup labels need their own controls. Do not reuse this gate as a universal policy engine.

Close

Keep the free lane inside the workshop boundary. Let a separate oracle sign the ship report.

If a workshop lane still helps, confirm the current docs first. Try free model access only for drafts you can discard.