A nightly rerun is useful only when the harness can prove which lane produced the score. A free server makes generation cheap, and cheap generation tempts people to overwrite yesterday's baseline with today's convenience sample. That overwrite hides real regressions, because a different endpoint is not the same experiment as the one you scored last week. The practical fix is a two-lane record: smoke may explore, and only adjudication may move the stored baseline.
Treat the golden set as a locked notebook, not as a chat window you can refresh. Each row needs a case id, a fixture payload, a grader version, and a provider fingerprint taken from the actual response. If any of those fields change, the new row is a different measurement, even when the prompt text looks identical. Mixing those measurements is how a quiet model swap gets reported as a product failure by the weekly chart.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project that currently offers free model access and a free server option. This draft does not name models or quote a token allowance, because those details were not verified against a primary source. Hardware, duration, and permanence are also omitted, so read the current project docs before you depend on either offer.
The free options matter in one narrow place, which is hosting the smoke lane for frequent regeneration. You can regenerate answers often enough to notice drift without spending a paid key on every pass. They should not host the adjudication lane unless the endpoint class, model id, and decoding settings match the baseline. A bargain rerun is a scout rather than a judge, and shared score files turn scout noise into team memory.
The artifact below is a proposal, and it was not run against a live endpoint for this draft. It stores a fingerprint, grades with a deliberately small keyword check, and refuses to promote a smoke row over an adjudication baseline. Replace the keyword check with your real grader, but keep the promotion rule intact across every rewrite. The promotion rule is the part that stops a free rerun from rewriting the stored history.
# Proposal only: not executed against a live server in this draft.
import hashlib
import json
from dataclasses import asdict, dataclass
ADJUDICATION = "adjudication"
SMOKE = "smoke"
GOLDEN = [
{
"case_id": "refund-window",
"required": ["14 days", "receipt"],
"fixture": "Customer asks for a refund on day 10 and has a receipt.",
},
{
"case_id": "no-receipt",
"required": ["cannot verify", "receipt"],
"fixture": "Customer asks for a refund and cannot produce a receipt.",
},
]
@dataclass(frozen=True)
class Fingerprint:
lane: str
endpoint_class: str
model_id: str
temperature: float
grader_version: str
def fingerprint_key(fp: Fingerprint) -> str:
raw = json.dumps(asdict(fp), sort_keys=True, separators=(",", ":"))
return hashlib.sha256(raw.encode()).hexdigest()[:16]
@dataclass
class Row:
case_id: str
fingerprint: Fingerprint
grade: str
reason: str
def grade_keywords(answer: str, required: list[str]) -> tuple[str, str]:
if not answer or not answer.strip():
return "schema_error", "empty completion"
missing = [k for k in required if k.lower() not in answer.lower()]
if missing:
return "fail", "missing=" + ",".join(missing)
return "pass", "keywords present"
def may_promote(incoming: Fingerprint, stored: Fingerprint | None) -> bool:
if incoming.lane != ADJUDICATION:
return False
if stored is None:
return True
return fingerprint_key(incoming) == fingerprint_key(stored)
def quality_rate(rows: list[Row]) -> float | None:
judged = [r for r in rows if r.grade in ("pass", "fail")]
if not judged:
return None
return sum(r.grade == "pass" for r in judged) / len(judged)
def compare(baseline: list[Row], incoming: list[Row]) -> dict:
base = {r.case_id: r for r in baseline}
matched = mismatches = flips = 0
for row in incoming:
prev = base.get(row.case_id)
if prev is None:
continue
same = fingerprint_key(row.fingerprint) == fingerprint_key(prev.fingerprint)
if not same:
mismatches += 1
continue
matched += 1
if row.grade != prev.grade:
flips += 1
return {
"matched_fingerprints": matched,
"fingerprint_mismatches": mismatches,
"grade_flips": flips,
}
Read the model id from the response body, and pass that string into Fingerprint without renaming it. Set endpoint_class to a label you control, such as free_server or pinned_provider, and keep the secret base URL outside the score file. Temperature belongs in the fingerprint because a 0.7 sample is not comparable to a 0.0 sample from the same prompt. The grader version belongs there too, because a stricter checker can fake a regression even when the model did not move.
Promotion is intentionally blunt, and a smoke row may be stored, charted, and alerted on without touching the baseline. The may_promote function returns false whenever the incoming lane is not adjudication, even if every keyword matched. A free-server batch can fill a side file all night and still leave the baseline file untouched for the next review. You investigate a smoke drop by rerunning those case ids on the adjudication lane, not by editing the baseline JSON.
If the adjudication rerun also drops, you have a candidate regression that deserves a diff of the completions. If the adjudication rerun holds, the smoke drop was a lane change and should be labeled as such. A synthetic walkthrough, labeled as unexecuted, shows why the split changes the decision you would otherwise publish. Imagine forty golden cases and an adjudication baseline of thirty-six passes, two fails, and two schema errors.
The quality rate ignores schema errors and is thirty-six over thirty-eight, or about 0.947 on that baseline. A later smoke batch on a free server returns thirty passes, six fails, and four schema errors. That smoke rate is thirty over thirty-six, or about 0.833, which looks like an eleven-point drop. Promoting that batch would announce a drop that the locked lane never confirmed, so the harness should leave 0.947 in place.
The commands below are a proposal, and you should substitute your own paths without committing secrets. Set the lane and the endpoint class in the environment, and leave the model id empty until the response fills it. Run the smoke output to a side file, then compare it with an explicit refusal to promote. A compare that defaults to promote will undo the separation the rest of the harness tried to build.
# Proposal commands. Substitute paths locally; do not commit secrets.
export EVAL_LANE=smoke
export EVAL_ENDPOINT_CLASS=free_server
export EVAL_MODEL_ID="" # fill from the response, never from memory
python harness.py --cases cases/golden.jsonl --out runs/smoke.jsonl
python harness.py --compare runs/baseline.jsonl runs/smoke.jsonl --promote never
The compare step should print three counts: matched fingerprints, lane mismatches, and grade flips inside a matched fingerprint. Only the third count is a regression candidate, and the other two counts explain why a chart moved. A lane mismatch is a scheduling fact, and a schema error is an instrumentation fact rather than a quality fact. Folding either count into the quality rate will train the team to chase the wrong chart during review.
You can wire the smoke lane to whatever free model access you currently have, provided the response returns a model id. The cron entry should fail closed when the response omits a model id, because the fingerprint is then incomplete. An incomplete fingerprint must not be compared as if it were pinned to last week's adjudication run. Log the HTTP status and the case id, then skip scoring, since a skipped row is more honest than a guessed model name.
This approach is a poor fit for several teams, and the limits are part of the method rather than a footnote. Do not use it if your baseline must blend providers on purpose, because the promotion rule will block that blend by design. Do not use it for safety, medical, or compliance judgments, because a keyword grader does not measure harm or policy fit. Do not use it if the free endpoint's terms forbid storing prompts or completions, because the score file is itself a store.
Do not use it if you cannot keep the adjudication endpoint stable for the life of the baseline you intend to defend. Without a stable adjudication lane, the promotion rule has nothing trustworthy to protect, and every rerun is smoke. Keyword checks also miss meaning changes that still contain the required words, so they are a gate rather than a verdict. Pair them with a second grader on a small stratified sample, and treat disagreement as a review queue rather than an automatic fail.
If you adopt one habit, adopt the refusal to promote, because cheap generation is worth having only when it cannot inherit pinned authority. A free server is a reasonable place to regenerate fixtures while you debug a grader or widen a smoke sample. Check the current MonkeyCode documentation if you want that smoke lane on their stated free model access. Keep the adjudication key and the baseline file on the lane you can actually reproduce next month.
Top comments (0)