Pin the Fixture, Not the Vendor: A Take-Home for Swap-Safe Eval Runners

A second reviewer cannot trust a take-home that only passes on one laptop and one model string. The score that survives a handoff is the fixture hash, a shared decision schema, and invariants that do not name a vendor. This packet asks a candidate to build a small evaluation runner for a frozen support-triage task. The runner must pass locally with a stub, then accept an optional live endpoint without edits to the assertions.

Free model access and a free hosted fixture matter here only because they let a grader replay the same bytes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims stay operator-supplied for this exercise, and they are not treated as measured capacity. No token quota, hardware profile, region list, or permanence promise appears here, because no primary source dated this draft supplied those figures.

The defect under test

Swap drift is the defect this packet isolates, and it shows up when a suite is green only on the draft-day endpoint. A candidate's checks then fail after the base URL, the model id, or the fixture host changes. Reviewers often mislabel that failure as an outage, even though the assertions were coupled to one vendor envelope. This packet leaves aside retries, outbox timing, and sticky flags, because those stories already have their own grading sheets.

The practical cost appears when hiring loops try to compare candidates without a shared document or a shared port. Two reviewers cannot score fairly when each demo depends on a private key and a laptop process that sleeps. A hosted fixture with a recorded hash removes that excuse for the document that both reviewers must read. A swappable adapter removes the matching excuse for the completion path, which keeps the handoff off one machine.

What the candidate receives

The prompt should stay on one page and should name files rather than a preferred brand. Candidates receive the six constraints below, together with a written note that any live call stays optional. A missing key must skip the network path and must still produce a complete stub report. The page should not mention a numeric allowance, because that number is not part of the contract under test.

Prompt constraints

1. Load the local fixture.json file, call one adapter, and write report.json with one decision object.
2. Read MODEL_BASE_URL, MODEL_API_KEY, and FIXTURE_URL only from the process environment, never from committed files.
3. Store the SHA-256 of the fixture bytes, and fail closed when fetched bytes differ from the pinned hash.
4. Enforce three invariants: citations use only fixture ids, unknown fields are dropped, and one idempotency key yields one decision id.
5. Ship a stub mode that needs no network, and a hosted mode that fetches the same fixture document.
6. State every claim left unverified, including allowance, retention, and region, instead of writing those claims into tests.

A candidate may point the live path at MonkeyCode's operator-stated free model route and may publish the fixture through its free server option. The grader must still pass when that live path is skipped for lack of a key. A numeric allowance must not appear in an assertion, a fixture field, or a comment that later tests treat as fact. When current project notes describe that route, the candidate cites them in the README and omits secrets.

Rubric for the reviewer

Behavior remains the only scoring surface that reviewers accept for this slice of the hiring loop. Prose about a product does not add points, and a green screenshot without a hash does not count. Reviewers should mark the live path as skipped whenever that path did not actually run in the session. They should not upgrade a stub-only pass into a written statement about any particular model route.

Score sheet rules

Check	Pass condition	Fail condition
Fixture pin	Report stores SHA-256 of the exact bytes	Assertion searches for a model id
Adapter swap	Stub and live modes share one schema	Live mode uses a second parser
Scope fence	Unknown fields are rejected or dropped	Extra fields are copied into the decision
Idempotency	Same key keeps the same decision id	Each call mints a fresh id
Secret handling	Key is read from the environment only	Key or token is written into the report
Claim hygiene	Unverified allowances are omitted	A fixed quota is asserted as fact

Four passing checks still count as a useful signal within this narrow slice of take-home work. A missing fixture pin is an automatic fail, because every other check can look green against the wrong document. The table is a grading aid, not a claim that any hosted route stayed up during a measured window.

Proposed sample, not a recorded run

The following Python is a proposed sample that reviewers may adapt inside their own hiring packet. It was not executed for this article, and it is not a latency or cost benchmark. The live call stays optional so that the stub path remains the deterministic grade for reviewers. Candidates who add a real HTTP client should keep the same decide function for both modes.

import hashlib, json, os, urllib.request
from pathlib import Path

def load_fixture(url: str | None, local: Path) -> bytes:
    if not url:
        return local.read_bytes()
    with urllib.request.urlopen(url, timeout=10) as resp:
        return resp.read()

def decide(fixture: dict, completion: str) -> dict:
    allowed = {item["id"] for item in fixture["items"]}
    ids = [tok.removeprefix("id:") for tok in completion.split() if tok.startswith("id:")]
    extra = set(fixture.get("echo_fields", [])) - {"id", "title"}
    if extra:
        return {"ok": False, "reason": "unknown_field", "decision_id": fixture["key"]}
    if any(i not in allowed for i in ids):
        return {"ok": False, "reason": "out_of_scope", "ids": ids, "decision_id": fixture["key"]}
    return {"ok": True, "reason": "in_scope", "ids": ids, "decision_id": fixture["key"]}

def run(base: Path) -> dict:
    raw = load_fixture(os.environ.get("FIXTURE_URL"), base / "fixture.json")
    pinned = os.environ.get("FIXTURE_SHA256")
    digest = hashlib.sha256(raw).hexdigest()
    if pinned and pinned != digest:
        raise SystemExit("fixture pin mismatch")
    fixture = json.loads(raw)
    completion = os.environ.get("STUB_COMPLETION", "use id:item-1 only")
    report = {
        "fixture_sha256": digest,
        "live_configured": bool(os.environ.get("MODEL_BASE_URL")),
        "decision": decide(fixture, completion),
    }
    (base / "report.json").write_text(json.dumps(report, indent=2))
    return report

if __name__ == "__main__":
    run(Path("."))

A matching fixture should stay small and stable, with no extra whitespace before the pin is recorded. Reviewers hash these exact bytes, and they should not reformat the file before pinning it. The sample completion cites item-1, and that identifier is the only one the fixture marks as allowed.

{
  "key": "case-17",
  "items": [
    {"id": "item-1", "title": "reset token flow"}
  ]
}

Commands a reviewer can reuse

Stub path

These commands stay illustrative, and they assume that both files already sit in the working directory. They do not call a live vendor, and they do not require a paid seat or a named model. A reviewer runs the pin check first, then the out-of-scope check, and only then considers a hosted URL.

python -m json.tool fixture.json >/dev/null
export FIXTURE_SHA256=$(python -c "import hashlib,pathlib; print(hashlib.sha256(pathlib.Path('fixture.json').read_bytes()).hexdigest())")
STUB_COMPLETION="use id:item-1 only" python runner.py
python -c "import json,os; r=json.load(open('report.json')); assert r['fixture_sha256']==os.environ['FIXTURE_SHA256']"
STUB_COMPLETION="use id:item-9" python runner.py
python -c "import json; r=json.load(open('report.json')); assert r['decision']['reason']=='out_of_scope'"

Hosted path

A hosted fixture check should fail closed when its URL is wrong or otherwise unreachable today. The invalid host below is deliberate, and a cached report from yesterday must not hide that failure. Replacing that URL remains a reviewer step, and it must never rewrite the assertion text itself.

FIXTURE_URL="https://example.invalid/fixture.json" python runner.py
echo "exit=$?"

If an operator has currently documented a free server, the reviewer fetches the fixture, recomputes the hash, and stores both values on the score sheet. Test names should stay generic so they do not embed a product name or a numeric quota. The same score sheet notes whether MODEL_BASE_URL was set, and it never prints the key.

Hidden cases to plant after the draft

The written prompt implies the cases below, but it should not reveal the exact hidden inputs. That gap is the place where swap drift usually hides during a candidate's quick local self-check. Graders should add those hidden cases only after the candidate has frozen every submitted file.

• A trailing newline in the fixture must change the hash and must fail the pin check.
• A completion citing id:item-9 must return out_of_scope when only item-1 exists in the file.
• A second run with the same fixture key must reuse decision_id rather than allocate another one.
• An empty MODEL_API_KEY must skip the live call and must not log the literal None value.
• A fixture that carries echo_fields outside id and title must not copy those fields into report.json.
• A live envelope with a different wrapper must still normalize to the same decision schema as the stub.

Candidates who snapshot one lucky completion will fail the hidden id case on the second run. Candidates who treat a free URL as a permanent dependency will fail the pin when that host is absent. Availability during the review window is an operator statement, not a property the suite should hardcode. The hidden list therefore functions as a contract test, not as a probe of answer style.

Failure modes before any model debate

Three ordinary contract failures show up long before reviewers argue about generated answer quality at all. The first pins a model id inside a unit test, so a free route with another id cannot pass. The second fetches the fixture once, then replaces it with a paraphrase, so reviewers no longer share bytes. The third writes an unverified allowance into a comment or assertion, which turns a changing commercial detail into a false invariant.

A fourth failure is a quiet adapter split between the local stub and the live response parser. The stub returns a plain string, while the live parser expects one vendor envelope and no other shape. The suite stays green only on the stub, and the candidate reports the swap as an infrastructure outage. The rubric should mark that split as an adapter defect rather than as evidence about route uptime.

A fifth failure prints the API key into report.json while the candidate chases a debug log. Secret handling fails in that case even when every invariant holds and the hash matches. Reviewers should delete that report from shared storage and then ask for a clean rerun. That rerun still uses the same fixture pin, so the candidate cannot quietly swap documents between attempts.

Who should not use this packet

This exercise does not rank any models, measure latency, or estimate spend on a live route. It does not permit hiring-packet text, customer records, or secrets to be published on a public fixture host. Teams that require a named licensed model or an air-gapped network should keep only the stub path. A formal vendor security review is also outside this packet, so those teams should skip the hosted path.

Operators should hand each candidate the current project documentation on the morning of the hiring loop. If that documentation disagrees with this packet, the documentation wins and the prompt should be patched. The packet only requires a swappable route and fixture bytes that remain fetchable during the review window. It should not be reused as a press claim about capacity, uptime, or a standing token grant.

Closing the review

The reviewer hosts the fixture, records the URL with the hash, and runs the stub path first. The live path runs only when a key is present and the endpoint answers within the exercise window. The score sheet must clearly record whether the live path was skipped during that review session. Readers who need this split can follow current operator notes on free model access and a free server, with assertions left on the hash.