Expire the Eval or Don't Enable: A Fail-Closed Measurement Checklist

A model-backed endpoint is not ready because the unit suite is green. It is ready when the measurement that can stop enablement is named, hashed, dated, owned, and fail-closed. If that measurement is missing, you are not shipping a feature. You are hoping.

This is not another reminder that tests exist. You already have those. This is the gate that decides whether a prompt change, a retrieval tweak, or a tool-schema edit may take production traffic.

What you are actually gating

Unit tests check deterministic code. Model-backed paths are not deterministic in the way your CI likes. You still need a number. You still need a cutoff. You still need a clock on the set that produced that number.

Without a clock, last quarter's goldens keep blessing this week's behavior. That is how a "measured" system drifts into an unmeasured one. Quietly.

You will copy four things:

1. A fail-closed rule.
2. A gate matrix you can paste into a PR template.
3. A YAML manifest the job must refuse to run without.
4. A small runner that exits non-zero when the evidence is incomplete.

None of this requires a vendor. A free model path and a spare server help when you want the job to run without burning a paid key. They are optional. The gate is not.

The fail-closed rule

If any of the following is true, the enablement does not happen. No silent skip. No "we'll backfill the eval."

• The eval set has no eval_id.
• The eval set has no content hash recorded in the repo.
• The eval set is older than its own max_age_days.
• The eval set has no human owner.
• The job cannot load the set from the path the manifest names.
• The score is below threshold and there is no waiver file with an expiry.
• The waiver is past expiry, or it does not name a ticket.

Read that again. A missing eval is a failed eval. An expired eval is a failed eval. An owned-by-nobody eval is a failed eval.

You do not "degrade to best effort" on the measurement that authorizes traffic. Best effort is for retries. Measurement is for permission.

Missing is red, not yellow

Teams love a skip. Skip looks kind in a dashboard. Skip is how a prompt PR lands on Friday and the goldens catch up "next sprint." Next sprint is where stale answers become product behavior.

If the job cannot run, the flag stays off. That is the whole method. Write it on the PR template so nobody has to remember it under pressure.

Gate matrix you can copy

Paste this into the PR that touches prompts, tools, retrieval, or the model-backed route.

Gate	Evidence you must attach	Fail-closed if missing
Identity	eval_id in eval_gate.yaml	Enablement blocked
Integrity	SHA-256 of the dataset file	Hash mismatch or empty hash
Freshness	generated_at + max_age_days	Age exceeded
Ownership	GitHub handle or team alias	Owner blank
Execution	CI log of the runner, same commit SHA	Job skipped or ran on another SHA
Threshold	min_pass_rate compared to scored output	Score absent or below cutoff
Waiver	waivers/<eval_id>.md with ticket + expiry	Waiver missing, expired, or unnamed
Rollback	Kill switch / flag name in the PR	No named flag, no enablement

You do not need all eight to be fancy. You need all eight to be present. Fancy comes later.

Treat the matrix as a receipt, not a vibe. If a cell is empty, the row failed. If the row failed, the flag stays down.

Manifest the job must refuse to invent

Create eval_gate.yaml next to the route you are protecting. Keep it boring.

# Proposal: commit this beside the model-backed route.
# Unexecuted example — wire paths to your repo.
eval_id: checkout_assist_v3
dataset_path: evals/checkout_assist_v3.jsonl
dataset_sha256: REPLACE_WITH_REAL_SHA256
generated_at: "2026-09-14"
max_age_days: 14
owner: "checkout-ml"
min_pass_rate: 0.86
blocking: true
feature_flag: "checkout_assist_v3"
model_path_env: MODEL_ENDPOINT

Two notes. First: blocking: true means CI red is the product. If you set it false, you are collecting telemetry, not a gate. Second: model_path_env names an environment variable, not a model. You pin the path in deploy config. You do not hide it in a notebook.

Hash the dataset in CI so a silent edit cannot ride along:

# Label: local helper, run from repo root.
python - <<'PY'
from pathlib import Path
import hashlib, sys
p = Path("evals/checkout_assist_v3.jsonl")
if not p.is_file():
    sys.exit("eval dataset missing")
digest = hashlib.sha256(p.read_bytes()).hexdigest()
print(digest)
PY

If that digest does not match dataset_sha256, the job fails. You do not "update the hash later." You update it in the same PR that changed the set, or you do not change the set.

Refresh without laundering the gate

Goldens have to move. Domains shift. That is not a license to move the cutoff in the same diff as the prompt.

Use a split:

1. Freeze the old hash in git history. Do not rewrite it.
2. Add rows in a dataset-only PR. New generated_at. New hash. Same min_pass_rate.
3. Land prompt or tool changes in a second PR that must beat the already-merged set.
4. Raise min_pass_rate only after the new set has survived a week of blocking CI.

If you change the prompt, the set, and the cutoff together, you did not measure a regression. You authored a new story and called it a score.

Keep the files dull and findable:

evals/checkout_assist_v3.jsonl
evals/checkout_assist_v3.README.md
waivers/
eval_gate.yaml
eval_gate.py

The README names who labels a row, what "correct" means, and which ticket last rebuilt the set. If that file is empty, the owner field in YAML is theater.

Runner that fails closed

The runner below is a proposal. It does not call a vendor SDK. It scores a JSONL file of {id, input, expect} rows against a callable you already own. Wire your own client. Keep the control flow.

# eval_gate.py — proposal / unexecuted example
from __future__ import annotations

import hashlib
import json
import os
import sys
from datetime import date, datetime, timezone
from pathlib import Path

import yaml


def fail(msg: str) -> None:
    print(f"FAIL-CLOSED: {msg}", file=sys.stderr)
    raise SystemExit(1)


def load_manifest(path: Path) -> dict:
    if not path.is_file():
        fail(f"manifest missing: {path}")
    data = yaml.safe_load(path.read_text()) or {}
    for key in (
        "eval_id",
        "dataset_path",
        "dataset_sha256",
        "generated_at",
        "max_age_days",
        "owner",
        "min_pass_rate",
        "blocking",
        "feature_flag",
    ):
        if data.get(key) in (None, ""):
            fail(f"manifest field empty: {key}")
    return data


def assert_fresh(generated_at: str, max_age_days: int) -> None:
    born = date.fromisoformat(generated_at)
    age = (date.today() - born).days
    if age > int(max_age_days):
        fail(f"eval expired: age={age}d max={max_age_days}d")


def assert_hash(dataset: Path, expected: str) -> None:
    digest = hashlib.sha256(dataset.read_bytes()).hexdigest()
    if digest != expected:
        fail(f"dataset hash mismatch: got={digest}")


def assert_waiver_if_needed(eval_id: str, pass_rate: float, min_pass: float) -> None:
    if pass_rate >= min_pass:
        return
    waiver = Path("waivers") / f"{eval_id}.md"
    if not waiver.is_file():
        fail(f"score {pass_rate:.3f} < {min_pass} and no waiver")
    text = waiver.read_text()
    if "ticket:" not in text.lower() or "expires:" not in text.lower():
        fail("waiver missing ticket: or expires:")
    # Keep parsing strict and local. Do not fetch the ticket tracker here.


def score_rows(dataset: Path, call) -> float:
    rows = [json.loads(line) for line in dataset.read_text().splitlines() if line.strip()]
    if not rows:
        fail("dataset empty")
    hits = 0
    for row in rows:
        got = call(row["input"])
        # Exact match on purpose. Fuzzy scoring belongs in a named scorer file.
        if str(got).strip() == str(row["expect"]).strip():
            hits += 1
    return hits / len(rows)


def main() -> None:
    manifest = load_manifest(Path("eval_gate.yaml"))
    if not manifest["blocking"]:
        fail("blocking is false; this job is not a gate")
    dataset = Path(manifest["dataset_path"])
    if not dataset.is_file():
        fail(f"dataset missing: {dataset}")
    assert_hash(dataset, manifest["dataset_sha256"])
    assert_fresh(manifest["generated_at"], manifest["max_age_days"])
    if not os.environ.get(manifest.get("model_path_env", "MODEL_ENDPOINT")):
        fail("model path env is unset; refuse to score against an implicit default")

    def call(prompt: str) -> str:
        # Wire your client here. Do not silently fall back to a different model.
        raise RuntimeError("plug in your model client")

    pass_rate = score_rows(dataset, call)
    assert_waiver_if_needed(manifest["eval_id"], pass_rate, float(manifest["min_pass_rate"]))
    print(json.dumps({
        "eval_id": manifest["eval_id"],
        "pass_rate": pass_rate,
        "feature_flag": manifest["feature_flag"],
        "commit": os.environ.get("GIT_SHA", ""),
        "scored_at": datetime.now(timezone.utc).isoformat(),
    }, indent=2))


if __name__ == "__main__":
    main()

Notice what the runner does not do. It does not default the model path. It does not treat blocking: false as success. It does not accept an empty dataset as 100%. Those are the holes teams leave "for later." Later is how stale goldens become production truth.

Exact match is a feature here. If your task is generative, do not sprinkle if "close enough" into this file. Add scorer.py, hash that file too, and name it in the manifest. Hidden fuzzy logic is a second model you forgot to version.

CI shape, not CI poetry

Keep the workflow stupid. One job. Same commit SHA as the merge.

# .github/workflows/eval-gate.yml — proposal
name: eval-gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      - "eval_gate.yaml"
      - "src/checkout_assist/**"
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python eval_gate.py
        env:
          MODEL_ENDPOINT: ${{ secrets.MODEL_ENDPOINT }}
          GIT_SHA: ${{ github.sha }}

If the path filter never matches your prompt files, the gate never runs. That is a hole. Put the prompt files where the filter looks, or drop the filter.

Print the JSON on success and archive it. A gate without a receipt is a story you tell in standup. You already know how those rot.

Store the receipt next to the flag name. When someone asks "why is v3 on?" you point at a commit SHA, a hash, a pass rate, and a date. You do not point at a Slack memory.

Where a free model path actually helps

You need a real MODEL_ENDPOINT for the runner. A laptop that is asleep is not a runtime. A paid key that someone rotated is not a runtime either.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you are already using MonkeyCode, the useful part here is mundane: free model access and a free server option give you a place to keep MODEL_ENDPOINT pointed at something that answers when CI asks. That is the whole integration. The checklist does not care which product serves the bytes. It cares that the path is explicit, the job is blocking, and the goldens can expire.

Do not point production traffic at the same endpoint you use to score goldens unless that is a decision you have written down. Scoring and serving are different jobs. Mixing them is how a load spike looks like an eval outage, or the reverse.

One practical split:

• CI eval uses the free server and free model access.
• Production serving uses the pinned path your deploy config already named.
• Both paths show up as env values. Neither is a default in code.

If you do not use MonkeyCode, run the same split against whatever endpoint you already trust. The artifact above still applies. Fork the checklist first. The product is optional.

Limitations

Exact string match will punish valid paraphrases. If your task is generative, replace score_rows with a named scorer file and version that file like the dataset. Do not hide the scorer in a prompt.

This checklist does not measure fairness, latency, or cost. If those can block a release, they need their own manifests. Do not overload min_pass_rate with three meanings.

max_age_days: 14 is an example, not a standard. Set the clock from how often your domain shifts. A tax table and a chitchat bot do not share a freshness budget.

The waiver path can become a culture hole. If every PR carries a waiver, you no longer have a gate. You have a diary. Cap concurrent waivers at one per eval_id, and make expiry shorter than max_age_days.

A hashed JSONL file is not a sampling strategy. If production traffic is 90% language A and your goldens are 90% language B, the gate is precise and wrong. Fix the set. Do not celebrate the score.

Who should not use this

Skip this if you do not have a model-backed route in production, and no plan to enable one. A checklist without a flag to protect is ceremony.

Skip this if your "eval" is a handful of vibe checks in a chat window. Write the JSONL first. Then gate.

Skip this if a human must approve every response at runtime. Your measurement problem is different. You need an audit log, not a pass rate.

Skip this if you cannot fail closed — regulated change windows, vendor-owned models you cannot pin, or an org that treats red CI as optional. The runner will only annoy you.

What you do on Monday

Pick one endpoint. One. Write eval_gate.yaml with a real hash or refuse to merge. Put blocking: true. Name the feature flag. Run the job on the PR that would have enabled it.

If the job cannot run, you do not enable. Stale goldens are not a vibe problem. They are an authorization problem. Treat them that way.