Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes

Shared inference hosts fail quietly when a probe set is missing, not when a dashboard still looks green. A five-probe floor card records the cheapest behaviors you still require after a host, prompt, or routing change. This workshop times that card at eighty-five minutes, with rerunnable files and a pass-or-fail ledger. Students leave with a JSON pack, a local scorer, and a printed floor they can recheck after every lab swap.

Why a floor card beats a chat anecdote

Anecdotes from one lucky prompt hide regressions that only appear on the second or third frozen case. Shared hosts also move under you, because capacity, routing, and hidden system prompts are not a public version pin. You need a frozen input set, a deterministic scorer, and a stored floor, or next week's run cannot be compared at all. The method below treats the model as an untrusted function with a tiny, documented surface.

Measurement talk in developer circles often outruns the tests that still discriminate. When a host starts returning fluent prose, empty objects, or a constant label, yesterday's demo stops being evidence. A floor card is deliberately small so a classroom can finish it, not so a vendor can be ranked. If the card cannot fail, it cannot teach anything useful about drift.

What this workshop is not

This is not a leaderboard, a latency study, or a claim about production model quality on a named system. It is a teaching lab that detects silent capability loss on one narrow, field-shaped task. If your team already runs a versioned eval platform with owners and an SLA, skip this outline and keep that platform. If you cannot freeze inputs because the task is open-ended prose, shrink the contract before you schedule the lab.

Workshop clock

Keep a visible timer for eighty-five minutes and refuse to expand the probe set during the first pass through the files.

1. 0–10 min — Frame the failure. Name one task, one JSON contract, and one host URL environment variable.
2. 10–25 min — Write the probe pack. Freeze five inputs, expected fields, and scoring modes on disk.
3. 25–45 min — Run the worked example. Score against a file fixture before any remote host is allowed.
4. 45–65 min — Exercise: point at a shared host. Keep the scorer identical and record a second floor card.
5. 65–80 min — Exercise: break one probe. Mutate a prompt or response shape and show the ledger turning red.
6. 80–85 min — Recap limits. List who should not ship this card as a quality gate.

Artifact layout

Students should create a directory they can zip, copy to another machine, and rerun without editing expects.

probe-floor/
  probes.json
  score_probes.py
  floor_card.json
  Makefile

The Makefile is the only class entrypoint, which stops ad-hoc flags from becoming the real curriculum.

.PHONY: fixture remote diff

fixture:
    python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json

remote:
    python3 score_probes.py --probes probes.json --base-url "$$PROBE_BASE_URL" --out floor_card.remote.json

diff:
    python3 score_probes.py --diff floor_card.json floor_card.remote.json

Probe pack schema

Keep scoring modes boring on purpose. Exact field match plus JSON parse success catch more host drift than a long prose rubric.

{
  "task": "semver_bump_from_diff",
  "contract": {
    "type": "object",
    "required": ["bump", "reason"],
    "properties": {
      "bump": {"enum": ["major", "minor", "patch", "none"]},
      "reason": {"type": "string", "minLength": 8, "maxLength": 160}
    }
  },
  "floor": {"min_pass": 5, "max_parse_fail": 0},
  "probes": [
    {
      "id": "P1_docs_patch",
      "input": {"diff_summary": "docs: fix typo in README install block"},
      "expect": {"bump": "patch"}
    },
    {
      "id": "P2_optional_field_minor",
      "input": {"diff_summary": "feat: add optional timeout_ms to ClientConfig"},
      "expect": {"bump": "minor"}
    },
    {
      "id": "P3_removed_field_major",
      "input": {"diff_summary": "breaking: remove ClientConfig.retry_count"},
      "expect": {"bump": "major"}
    },
    {
      "id": "P4_empty_diff_none",
      "input": {"diff_summary": ""},
      "expect": {"bump": "none"}
    },
    {
      "id": "P5_chore_none",
      "input": {"diff_summary": "chore: reformat imports with no behavior change"},
      "expect": {"bump": "none"}
    }
  ]
}

Five probes are a floor, not coverage, and they exist to fail closed when a host collapses. Watch for fluent prose, empty JSON, or a constant minor returned for every distinct case. If a pair wants a sixth probe during the first hour, park it in a notes file instead of changing the pack.

Worked example students can rerun

Label: this example is a local teaching fixture, not a measured vendor benchmark and not a claim about any hosted model. The file backend returns canned JSON so the scorer can be graded without a network round trip. Remote calls below use a lab default path; change that path to match whatever route your host actually documents.

# score_probes.py — teaching example, not a production eval platform
from __future__ import annotations

import argparse, json, sys, urllib.request
from pathlib import Path

SYSTEM = (
    "Return only JSON with keys bump and reason. "
    "bump must be major, minor, patch, or none."
)

def load_probes(path: Path) -> dict:
    return json.loads(path.read_text())

def complete_file(probe: dict) -> str:
    bump = probe["expect"]["bump"]
    return json.dumps({"bump": bump, "reason": f"fixture:{probe['id']}"})

def complete_http(base: str, probe: dict, timeout: float = 30.0) -> str:
    payload = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(probe["input"])},
        ]
    }).encode()
    req = urllib.request.Request(
        base.rstrip("/") + "/v1/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = json.loads(resp.read().decode())
    return body["choices"][0]["message"]["content"]

def score_one(contract: dict, probe: dict, raw: str) -> dict:
    row = {"id": probe["id"], "pass": False, "parse_ok": False, "detail": ""}
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        row["detail"] = "not_json"
        return row
    row["parse_ok"] = True
    if set(contract["required"]) - set(data):
        row["detail"] = "missing_keys"
        return row
    if data.get("bump") != probe["expect"]["bump"]:
        row["detail"] = f"bump:{data.get('bump')}"
        return row
    reason = data.get("reason", "")
    if not isinstance(reason, str) or not (8 <= len(reason) <= 160):
        row["detail"] = "reason_len"
        return row
    row["pass"] = True
    row["detail"] = "ok"
    return row

def run(probes: dict, base_url: str) -> dict:
    rows = []
    for probe in probes["probes"]:
        raw = (
            complete_file(probe)
            if base_url.startswith("file:")
            else complete_http(base_url, probe)
        )
        rows.append(score_one(probes["contract"], probe, raw))
    passed = sum(1 for r in rows if r["pass"])
    parse_fail = sum(1 for r in rows if not r["parse_ok"])
    floor = probes["floor"]
    return {
        "task": probes["task"],
        "passed": passed,
        "parse_fail": parse_fail,
        "floor_ok": passed >= floor["min_pass"] and parse_fail <= floor["max_parse_fail"],
        "rows": rows,
    }

def diff_cards(a: dict, b: dict) -> int:
    print(f"local_floor_ok={a['floor_ok']} remote_floor_ok={b['floor_ok']}")
    ids = {r["id"]: r for r in a["rows"]}
    rc = 0
    for row in b["rows"]:
        prior = ids.get(row["id"], {})
        if prior.get("pass") and not row["pass"]:
            print(f"REGRESS {row['id']} {prior.get('detail')} -> {row['detail']}")
            rc = 1
        elif prior.get("pass") != row["pass"]:
            print(f"CHANGE  {row['id']} pass {prior.get('pass')} -> {row['pass']}")
            rc = 1
    return rc

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--probes")
    p.add_argument("--base-url")
    p.add_argument("--out")
    p.add_argument("--diff", nargs=2)
    args = p.parse_args()
    if args.diff:
        a = json.loads(Path(args.diff[0]).read_text())
        b = json.loads(Path(args.diff[1]).read_text())
        return diff_cards(a, b)
    pack = load_probes(Path(args.probes))
    card = run(pack, args.base_url)
    Path(args.out).write_text(json.dumps(card, indent=2) + "\n")
    print(json.dumps({"floor_ok": card["floor_ok"], "passed": card["passed"]}, indent=2))
    return 0 if card["floor_ok"] else 2

if __name__ == "__main__":
    sys.exit(main())

Expected fixture command for every pair, before anyone exports a remote URL:

python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json

The teaching fixture always returns the expected bump, so floor_ok must be true before PROBE_BASE_URL is set. That order is the lab, not a ceremony around the lab.

Exercise 1 — freeze the contract, not the essay (15 minutes)

Students often want a long rubric because it feels more serious than five enum checks. Stop that impulse and ask each pair to delete any probe whose expect field is a free-text essay. A probe that cannot fail in one sentence is not frozen yet, and it will be edited to match a nicer model next week.

Checklist for the teaching assistant:

• Every probe has a stable id that will survive later wording edits.
• bump is an enum, never a list of allowed synonyms in natural language.
• reason is length-bounded so empty strings and novels both fail the same way.
• min_pass equals the probe count on day one; lowering it requires a written note.

Exercise 2 — change the host, not the scorer (20 minutes)

Export one URL and keep probes.json byte-identical. A local scorer should not care which process sits behind a compatible HTTP path, only whether the floor still holds. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can sit behind PROBE_BASE_URL once the file backend already prints floor_ok: true.

export PROBE_BASE_URL="http://127.0.0.1:8080"
make remote
make diff

Interpret the diff with a table, not with a vibe from a single chat window in another tab.

Diff signal	Meaning in this lab	Next action
floor_ok stays true	Floor still holds on this host	Record the card; do not add probes yet
parse failures greater than zero	Contract broke into prose or invalid JSON	Fix the prompt or reject the host
one bump mismatch	Task policy drifted on a frozen case	Keep the probe; do not edit expect
all bumps become minor	Classifier collapsed to a constant	Fail the floor; do not average scores

Do not invent a latency number, a token budget, or a quality rank from this table. The only honest outputs are floor_ok, per-probe detail codes, and whether a previously passing id regressed.

Exercise 3 — inject one honest failure (15 minutes)

Change the system string so it asks for Markdown fences, then rerun make remote without touching expects. The ledger should show not_json and floor_ok false on that host. If the card stays green, the scorer is too loose, and the rest of the workshop is invalid until the detail codes are strict again.

Optional mutation list, one change at a time:

• Truncate reason to three characters and expect reason_len.
• Force bump to the string Minor with surrounding spaces.
• Return two JSON objects concatenated in one message body.
• Echo the user input with no bump key at all.

Each mutation should map to a single detail code already printed by the scorer. Students who add a new code must document it beside the probe pack before they change Python.

Limitations

A five-probe floor card will not tell you that a host is good enough for production traffic. It only tells you that a host is still able to clear a tiny, frozen bar you wrote down in advance. Exact enum match is brittle if the real task is stylistic writing, multi-file refactors, or tool loops with side effects. Shared hosts can pass the card in the morning and fail it in the afternoon because routing is not a pin you control. This outline also assumes JSON contracts; if you cannot shrink the task to fields, do not fake a floor with a subjective one-to-five score.

The HTTP helper is a lab default, not a specification of any product. Timeouts, auth headers, response envelopes, and route names must follow the host you actually run. Do not paste secrets, customer diffs, or licensed source into probes.json just to make the cases feel realistic.

Who should not use this approach

• Teams that need contractual uptime, data-handling terms, or a named model revision they can cite.
• Classrooms that skip the file fixture and start on a public URL during the first half hour.
• Workloads with secrets, personal data, or licensed code inside the probe inputs.
• Anyone hoping five cases will replace unit tests, code review, or a staging environment.

Recap for the last five minutes

Write three lines on the board and stop talking over them.

1. Freeze inputs and expects on disk before you change hosts or prompts.
2. Score fields, not vibes, and fail closed on parse errors every time.
3. Treat a green floor card as permission to continue testing, not as proof of quality.

The same pack should rerun next week without editing expects to match a nicer answer. If you must change a probe, bump its id and record why the old floor died, because silent edits are how drift becomes folklore.