My Model Picker Only Checked That the Name Existed. Three Probes Said Otherwise.

A batch job I run every night classifies a few hundred short support messages into six fixed labels, which is boring work right up until the model decides to answer in prose. Last week I swapped the model on a Friday evening, watched the first fifty items parse cleanly, and went to bed feeling clever about it. By morning roughly a third of the output had come back wrapped in code fences, and my parser had quietly written None into every one of those rows. Why do I keep trusting a name in a list over a two-minute test that would have told me the truth?

So I stopped picking models by reputation and built a preflight that answers three narrow questions before a single batch item is spent. Does the route return parseable JSON that also obeys an unusual instruction, and can it follow a negative constraint I can check with a regex? Can it hold three sentinels across a long input, or does the middle quietly fall out of the window? And when the answer to any of those is unclear, does my pipeline treat that as a no instead of a maybe?

I ran the preflight against MonkeyCode's free model access on its free server option, because both are cheap enough that spending thirty-six calls before a batch feels reasonable rather than extravagant. I did not verify model names, quotas, or hardware, and I am treating free access and the free server option as operator-supplied availability claims rather than measured guarantees. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The probe design below is mine, and it stays useful with any client you already have.

The three questions I should have asked first

Each question maps to one probe, one cheap check, and one failure I can actually see in the output instead of guessing about.

• Structured output: can the route produce an object that parses, with values I did not hand it in the schema?
• Negative constraint: can it avoid a single letter across a whole sentence, which no amount of pattern-matching fluency fakes?
• Long-input recall: can it return tokens from the start, middle, and end of a padded prompt in the order I asked for?

The important design choice is that every probe embeds a value the model cannot guess. If a probe can pass by returning a plausible default, it is measuring your schema, not the model.

Probe 1: a schema can pass while the instruction fails

The first version of this probe asked for {"label": "ok"} and accepted anything that parsed as JSON. It passed on a route that replied {"label": "ok"} to every single prompt, which told me nothing except that the route knew the word ok. The fix is a canary string plus a count the model has to compute.

# probe.py — three capability probes, standard library only.
from __future__ import annotations

import json
import re
import time
from dataclasses import dataclass, asdict
from typing import Callable, Literal

PROBE_VERSION = "2026-10-26.1"
CANARY = "ZX41-quartz"
SENTINELS = ["S1-kestrel", "S2-basalt", "S3-nimbus"]

Status = Literal["PASS", "FAIL", "UNKNOWN"]
Call = Callable[[str], str]  # prompt -> reply text; you supply the client


@dataclass
class Result:
    route: str
    model: str
    probe: str
    status: Status
    reason: str
    slowest_s: float
    probe_version: str = PROBE_VERSION


def probe_json(call: Call) -> tuple[Status, str]:
    prompt = (
        "Reply with exactly one JSON object and nothing else: no prose, no fences.\n"
        'It must have the keys "canary" and "count".\n'
        f'Set "canary" to the exact string {CANARY}.\n'
        'Set "count" to the number of items in this list: alpha, beta, gamma.'
    )
    raw = call(prompt).strip()
    raw = re.sub(r"^```

(?:json)?|

```$", "", raw, flags=re.MULTILINE).strip()
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError as exc:
        return "FAIL", f"not json: {exc}"
    if not isinstance(obj, dict):
        return "FAIL", f"json but not an object: {type(obj).__name__}"
    if obj.get("canary") != CANARY:
        return "FAIL", f"canary ignored, got {obj.get('canary')!r}"
    if obj.get("count") != 3:
        return "FAIL", f"count wrong, got {obj.get('count')!r}"
    return "PASS", "object, canary and count all correct"

Note the fence stripping: I strip it so the probe reports what the model did, and then I count fence-wrapped replies separately as a warning. A route that needs stripping will break the day you change your parser.

Probe 2: negative constraints expose a different failure

A model that writes beautifully can still be incapable of avoiding one letter, and that asymmetry is exactly what a constrained rewrite job needs to know. This probe is a single sentence, a forbidden character, and a length floor so a one-word non-answer cannot sneak through.

def probe_negative(call: Call) -> tuple[Status, str]:
    prompt = (
        "Describe a red bicycle in one sentence. "
        "The sentence must not contain the letter e, upper or lower case. "
        "Reply with the sentence only."
    )
    raw = call(prompt).strip()
    if not raw:
        return "FAIL", "empty reply"
    if re.search(r"[eE]", raw):
        return "FAIL", "forbidden letter present"
    if len(raw.split()) < 4:
        return "FAIL", f"suspiciously short: {raw!r}"
    return "PASS", "negative constraint respected"

Probe 3: long input, three sentinels

Truncation and middle-dropping never announce themselves, and they look identical to a model that simply ignored you. Padding the prompt with filler and asking for the sentinels in reverse order separates the two, because a route that lost the middle cannot produce the second token at all.

def probe_echo(call: Call) -> tuple[Status, str]:
    filler = "The queue depth is sampled once per minute. " * 40
    prompt = (
        "The text below contains three sentinels. Reply with only those three "
        "sentinels in reverse order, separated by single spaces.\n\n"
        f"{SENTINELS[0]}\n{filler}\n{SENTINELS[1]}\n{filler}\n{SENTINELS[2]}"
    )
    got = call(prompt).split()
    want = list(reversed(SENTINELS))
    if got == want:
        return "PASS", "sentinels recovered in reverse order"
    return "FAIL", f"expected {want}, got {got[:6]}"

The runner and the two-run rule

One run per probe tells you a route can do something once, which is not the same as being routable for a nightly job. I run everything twice, serially, and I collapse the attempts conservatively.

PROBES = {"json": probe_json, "negative": probe_negative, "echo": probe_echo}


def run_matrix(route: str, model: str, call: Call, runs: int = 2) -> list[Result]:
    results = []
    for name, probe in PROBES.items():
        attempts = []
        for _ in range(runs):  # serialise: no threads, no cold-start blur
            t0 = time.monotonic()
            try:
                status, reason = probe(call)
            except Exception as exc:  # auth, DNS, 5xx, socket errors
                status, reason = "UNKNOWN", f"{type(exc).__name__}: {exc}"
            attempts.append((status, reason, time.monotonic() - t0))
        statuses = {a[0] for a in attempts}
        final: Status = (
            "PASS" if statuses == {"PASS"}
            else "FAIL" if "FAIL" in statuses
            else "UNKNOWN"
        )
        results.append(
            Result(route, model, name, final, attempts[0][1],
                   round(max(a[2] for a in attempts), 2))
        )
    return results


if __name__ == "__main__":
    import my_client  # your thin wrapper; credentials stay in env vars
    rows = []
    for route, model in [("free-server", "<model-a>"), ("laptop", "<model-b>")]:
        rows += run_matrix(route, model, my_client.make(route, model))
    print(json.dumps([asdict(r) for r in rows], indent=2))

Then read the matrix without opening Python again:

python probe.py > matrix.json
jq -r '.[] | [.route, .model, .probe, .status, .slowest_s] | @tsv' matrix.json | column -t

What broke in the first 48 hours

1. The canary-free probe passed everything. My first JSON probe accepted {"label": "ok"} on repeat, so I had a green matrix and a broken job. Adding an unguessable string and a count made the probe falsifiable.
2. The echo probe failed a route that was fine. I compared against the raw input, including whitespace, so every reply looked wrong until I normalised to tokens.
3. Concurrency blurred the results. Running probes in a thread pool on a shared free server mixed connection errors with real answers, so I serialised and accepted the slower run.
4. I probed inside the critical path. Thirty-six calls before the first batch item made the job feel broken, so the matrix now runs as a separate step that writes a file the job reads.

The routing ladder

A probe matrix is only useful if it changes a decision, so here is the table I actually follow.

json	negative	echo	Batch job?	What I do instead
PASS	PASS	PASS	Yes	Default route for that job shape
PASS	FAIL	PASS	Extraction only	No constrained rewrites, no style rules
FAIL	any	any	No	Manual spot-check first, chat-shaped tasks only
any	any	FAIL	No	Short prompts at most; assume the long window is a rumour
UNKNOWN	any	any	No	Check auth and network, then re-probe serially

The UNKNOWN row is the one that saved me the most time, because a connection error and a model refusal look identical in a log and imply completely different fixes.

A cassette so a restart replays instead of re-spending

Free tiers are free, but they are not unlimited, and a restart that re-sends every prompt is a bad trade. I append each call to a JSONL cassette keyed by a hash, so a rerun reads the recorded reply and only calls out for misses.

import hashlib


def cassette_key(route: str, model: str, prompt: str,
                 version: str = PROBE_VERSION) -> str:
    payload = "|".join([route, model, version, prompt])
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

Include the probe version in the key, or a tightened probe will happily replay an old answer and tell you your fix did nothing.

Limitations and who should skip this

Three probes are a smoke test, not an evaluation, and two runs tell you nothing about variance across a week. If you already have an eval harness with labelled data, this preflight adds little and belongs inside that harness instead. Teams with one pinned provider and a stable contract should spend the effort on output validation rather than route selection. And if your job cannot tolerate an extra thirty-six calls before it starts, run the matrix on a schedule and read a dated file rather than probing inline.

What I would repeat

I would keep the canary, the two-run rule, and the separate preflight step, and I would keep the matrix in the repository next to the job that depends on it. I would drop the cleverness of probing inside the run, and I would write the UNKNOWN branch before the first PASS branch, because that is the one that gets skipped. The cheapest part of this whole exercise was the probing, and the most expensive part was a night of None values in a table nobody checked.

If you want to try it, the probes above are client-agnostic; here they ran against MonkeyCode's free model access on the free server option, which made a full matrix cheap enough to run twice before a batch.