Swap the Model Before You Score

You promised a hire/no-hire by lunch. The zip is tidy, the GIF is flattering, and the README's first instruction is export OPENAI_API_KEY. You do not have that key in the interview budget. You also do not have the model they used at 1 a.m.

The recording is a souvenir. It is not a score.

That gap is the assignment. A candidate can look sharp on the model already sitting in their shell. You need to know whether the work still grades when you swap the model and the server under it. Models move. Keys expire. A take-home glued to one vendor is already stale when you unzip it.

Think of it like grading a cooking test after you change the oven. If the dish only works in their kitchen, you learned about their kitchen. You did not learn about their method. The public argument around evals keeps circling the same crack: the tests age faster than the systems they claim to measure. Your take-home should assume the swap, not apologize for it.

The assignment you actually send

The block below is a proposed take-home, not a war story from a named hiring loop. You freeze three things: the prompt set, the score schema, and the rule that the reviewer will change MODEL_NAME. The candidate ships a harness. You point that harness at a different base URL and you expect a score file, not a speech.

Keep PROMPT.md boring. Boring is load-bearing.

# Take-home: model-swap eval harness

Build a CLI that reads cases.json, calls a chat model through
MODEL_BASE_URL and MODEL_NAME, and writes score.json.

Rules:
- HTTP + JSON only. Do not import a vendor SDK by name.
- Temperature must be 0. The rubric grades structure, not vibes.
- score.json must validate against schema/score.schema.json.
- The same command must work after the reviewer changes MODEL_NAME.

You will be graded after we swap the model. If the zip only runs on
the model you used while writing it, the score is zero.

That last sentence is the job. Everything else is plumbing. If you bury the swap in a footnote, they will optimize for the GIF instead.

Put the rubric where a command can read it

A rubric that lives in a doc is a speech. A rubric that lives in rubric.yaml is an input. You want the second one. The harness should fail closed when a required field is missing, the same way a compiler fails closed when a type is missing.

# rubric.yaml
version: 1
max_score: 12
checks:
  - id: schema_valid
    points: 3
    how: score.json matches schema/score.schema.json
  - id: vendor_lock
    points: 3
    how: no vendor SDK import, no hardcoded host
  - id: swap_replay
    points: 4
    how: same cases.json, different MODEL_NAME, both runs emit scores
  - id: freeze
    points: 2
    how: temperature 0 and cases.json untouched

You do not need twelve dimensions. You need four that a stranger can re-run from disk. If a check cannot be decided without watching them talk, it does not belong in this take-home.

A sample harness worth pasting

This sample is labeled on purpose: it is an unexecuted example, not a benchmark and not a latency claim. It talks to whatever chat-completions endpoint you export. It does not know the product name of the model. That ignorance is the feature.

# eval.py
import json, os, sys, urllib.request
from pathlib import Path

BASE = os.environ["MODEL_BASE_URL"].rstrip("/")
NAME = os.environ["MODEL_NAME"]

def chat(prompt: str) -> str:
    body = json.dumps({
        "model": NAME,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/v1/chat/completions",
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]

def grade_reply(reply: str, case: dict) -> dict:
    expect = case["must_contain"]
    hits = [s for s in expect if s.lower() in reply.lower()]
    return {
        "id": case["id"],
        "pass": len(hits) == len(expect),
        "hits": hits,
        "missing": [s for s in expect if s not in hits],
    }

def main(cases_path: str, out_path: str) -> None:
    cases = json.loads(Path(cases_path).read_text())
    rows = [grade_reply(chat(c["prompt"]), c) for c in cases]
    passed = sum(1 for r in rows if r["pass"])
    Path(out_path).write_text(json.dumps({
        "model": NAME,
        "base_url": BASE,
        "passed": passed,
        "total": len(rows),
        "rows": rows,
    }, indent=2))

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

A reviewer should be able to run two commands and get two score files. Same cases. Different MODEL_NAME. If either command dies because a SDK is missing, they failed the lock check before you read a single answer.

export MODEL_BASE_URL=http://127.0.0.1:8080
export MODEL_NAME=reviewer-a
python eval.py cases.json score-a.json

export MODEL_NAME=reviewer-b
python eval.py cases.json score-b.json

python -m json.tool score-a.json >/dev/null
python -m json.tool score-b.json >/dev/null

Those hosts are placeholders. You will substitute the endpoint you actually review against. The shape of the command does not change. That is the whole trick.

Add a lock test so you are not grepping imports by hand at 11:50.

# test_no_vendor_lock.py
from pathlib import Path
import ast

BANNED = {"openai", "anthropic", "google.generativeai"}

def test_no_vendor_sdk():
    tree = ast.parse(Path("eval.py").read_text())
    names = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            names += [a.name.split(".")[0] for a in node.names]
        if isinstance(node, ast.ImportFrom) and node.module:
            names.append(node.module.split(".")[0])
    locked = BANNED.intersection(names)
    assert not locked, f"vendor SDK imported: {locked}"

def test_base_url_not_hardcoded():
    src = Path("eval.py").read_text()
    assert "api.openai.com" not in src
    assert "MODEL_BASE_URL" in src

Run pytest test_no_vendor_lock.py. If it fails, you can stop. The rest of the rubric is commentary on a zip that already confessed.

How the four checks feel in a real review

Schema valid is the easy one. Either score.json opens or it does not. Treat a pretty Markdown report with no JSON as a miss. You asked for a file a script can read. They sent you an essay.

Vendor lock shows up as a missing wheel. They wrote from openai import OpenAI because that is what their weekend tutorial used. The import is a confession. It says the take-home was built around a brand, not around a contract.

Swap replay is where most zips die. The first model returns a short JSON-ish blob and their grader looks for a key. The second model wraps the same answer in a sentence, and the grader panics. That panic is useful. It tells you they graded a dialect, not a result. A harness that only understands one model's manners is a client for a pet, not a tool.

Freeze is the quiet check. If cases.json was edited to match answers they already saw, you are not measuring a model. You are measuring a spoiler. Hash the cases file in the zip you sent and compare. If the hash moved, the candidate moved the test.

Where a free model and a free server actually help

You still need a place to point MODEL_BASE_URL that does not start an invoice during review. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough to replay a frozen prompt set and write score.json twice. That is the entire use. It does not make the candidate faster in production. It does not replace a staging cluster. It lets you finish the swap without asking finance for a key.

If the zip cannot tolerate that swap, do not argue with the GIF. The GIF was never the artifact.

Failure modes you will see by Thursday

The SDK-shaped hole is only the first. The second is scoring prose. They ask the model to "explain the bug," then they grep for the word race. A model that says "data race" passes. A model that says "concurrent mutation of the same counter" fails. Same diagnosis, different costume. Make the case require a stable token, like the test name or an error code, not an adjective.

The third is temperature as personality. They leave temperature at 0.7 because the demo "felt alive." Alive is not a rubric. Variance is a bug when you are trying to compare two models on the same cases.

The fourth is the notebook that only runs if you click top to bottom while logged into their cloud. A reviewer is not their laptop. If the path to score.json is a ritual, the take-home is a performance.

The fifth is silent truncation. The harness catches urlopen errors and writes an empty rows array with passed: 0. That looks like a model failure. It is a network failure wearing a score. Fail loud. Print the HTTP status. Exit non-zero. Do not launder transport errors into "the model is dumb."

What this does not measure

This take-home will not tell you if they can sit in an incident channel. It will not tell you if they can read a profiler. It will not tell you whether they are kind in code review. If the role is "make our one vendor eval suite two percent better," a model-swap assignment is the wrong oven. Give them that vendor and grade that vendor.

Free model access is also not an SLO. Endpoints change. Rate limits exist even when nobody quotes a number here. Do not write the job post as if a review-time server is a production dependency. Do not treat a candidate's latency screenshot as evidence unless you measured it on hardware you control. This article does not include those numbers, because they were not measured here.

Skip the pattern for screens where you only need a short coding signal. Fizzbuzz does not need a model swap. Agent work that will outlive one vendor does.

Ship the swap instructions with the prompt

Put PROMPT.md, rubric.yaml, cases.json, and the two-command replay in the same zip you send. Tell them, in one sentence, that you will change MODEL_NAME before you score. People optimize for the test they can see. If they cannot see the swap, they will hardcode the weekend.

When score-a.json and score-b.json both exist, you are grading a method. When only the GIF exists, you are grading a subscription. Swap the model. Then read the zip.