Probe Both Ends Before You Route

A routing rule is only as good as its worst measurement.

Local-first means the local runtime is the default. Default is not the same as correct.

Two runtimes never behave alike. A local model and a free remote server drift apart quietly. Versions change, limits change, and free tiers can change without a changelog.

Your parser will not complain about the drift. It will coerce, default, or truncate instead.

So measure both ends with one harness. Store the result as data. Let the router read the data.

That is the whole method, and the rest of this post is the artifact.

Five checks, one harness

The harness targets any OpenAI-compatible chat endpoint. It runs the same prompts against local and remote. It writes one JSON card per endpoint.

Here is the call wrapper. It is a sketch, not a benchmark suite.

# probe.py — design sketch. Run it against your own endpoints.
# Pin your own model ids; do not hardcode names you have not tested.
import json, time, statistics, datetime, urllib.request

FILLER = "The archive log records routine events. " * 400

def call(url, key, model, prompt, stop=None, temperature=0.0):
    body = {"model": model, "temperature": temperature,
            "messages": [{"role": "user", "content": prompt}]}
    if stop:
        body["stop"] = stop
    req = urllib.request.Request(
        url, data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=60) as r:
        out = json.loads(r.read())
    return out["choices"][0]["message"]["content"], (time.perf_counter() - t0) * 1000

Check one is strict schema compliance. Ask for three fields, parse the reply, count violations without repairing anything.

SCHEMA_PROMPT = ('Reply with JSON only: '
                 '{"id": int, "label": str, "ok": bool}. No prose.')

def check_json(url, key, model, n=3):
    good = 0
    for _ in range(n):
        text, _ = call(url, key, model, SCHEMA_PROMPT)
        try:
            obj = json.loads(text)
            good += int(set(obj) == {"id", "label", "ok"})
        except json.JSONDecodeError:
            pass
    return good / n

Check two is stop-sequence honor. Vendors differ on whether the stop string itself is echoed. Read your own output once and pin the rule.

def check_stop(url, key, model):
    text, _ = call(url, key, model, "Count: one two three four five.", stop=["three"])
    return "three" not in text and "four" not in text

Check three is instruction retention at depth. Bury a unique token at 25 percent and again at 75 percent of filler text.

TOKEN = "ZQ7-KESTREL"

def check_needle(url, key, model, position):
    cut = int(len(FILLER) * position)
    prompt = FILLER[:cut] + TOKEN + " " + FILLER[cut:] + "\nReply with only the token."
    text, _ = call(url, key, model, prompt)
    return TOKEN in text

Check four is repeat stability at temperature zero. Identical prompts should not produce different structures.

def check_stability(url, key, model, prompt, n=3):
    outs = {call(url, key, model, prompt)[0].strip() for _ in range(n)}
    return 1.0 if len(outs) == 1 else 0.0

Check five is a latency floor. Five sequential calls give a weak but honest shape of the distribution.

def check_latency(url, key, model, n=5):
    times = [call(url, key, model, "Reply with the word ready.")[1] for _ in range(n)]
    return {"median_ms": round(statistics.median(times)),
            "p90_ms": round(sorted(times)[-1])}

Every card carries a timestamp. A measurement without a date is a rumor.

def card(name, url, key, model):
    return {
        "endpoint": name,
        "checked_at": datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"),
        "model": model,
        "json_strict": check_json(url, key, model),
        "stop_honor": check_stop(url, key, model),
        "needle_25": check_needle(url, key, model, 0.25),
        "needle_75": check_needle(url, key, model, 0.75),
        "repeat_stability": check_stability(url, key, model, SCHEMA_PROMPT),
        "latency": check_latency(url, key, model),
    }

python probe.py --name local  --url "$LOCAL_URL"  --key "$LOCAL_KEY"  --model "$LOCAL_MODEL"  > cards/local.json
python probe.py --name remote --url "$REMOTE_URL" --key "$REMOTE_KEY" --model "$REMOTE_MODEL" > cards/remote.json

A plausible card looks like this. The numbers below are placeholders, not measured results.

{
  "endpoint": "local",
  "checked_at": "2026-10-16T09:12:04+00:00",
  "json_strict": 1.0,
  "stop_honor": true,
  "needle_25": true,
  "needle_75": false,
  "repeat_stability": 1.0,
  "latency": {"median_ms": 640, "p90_ms": 1180}
}

Route by check result, not by machine size

The interesting cases are mixed. A small local model can win the schema check and lose the depth check. A large remote model can win depth and lose stability.

Here is a router that reads only cards. It fails closed when a card is missing or stale.

MAX_AGE = datetime.timedelta(days=3)

def fresh(card, now):
    age = now - datetime.datetime.fromisoformat(card["checked_at"])
    return age <= MAX_AGE

def route(task, cards, now):
    local, remote = cards.get("local"), cards.get("remote")
    if not local or not fresh(local, now):
        raise RuntimeError("no trustworthy local card")
    if task.wants_json:
        if local["json_strict"] == 1.0 and local["repeat_stability"] == 1.0:
            return "local"
        if remote and fresh(remote, now) and remote["json_strict"] == 1.0 \
           and remote["repeat_stability"] == 1.0 and not task.has_secrets:
            return "remote"
        raise RuntimeError("no endpoint meets the structured contract")
    if task.prompt_tokens > 4000 and not local["needle_75"]:
        if remote and fresh(remote, now) and remote["needle_75"] and not task.has_secrets:
            return "remote"
    return "local"

Three rules do most of the work. A stale card is treated like no card. Secret-bearing tasks never leave, whatever the card says. Local stays the default for everything the probes do not justify moving.

A capability card is a passport with an expiry date. An expired passport does not prove you cannot travel. It proves nobody checked recently.

Consider a common failure. A remote endpoint passes the schema check but fails stability. Two of three runs append an extra note field. The parser accepts both shapes. A downstream dedupe keyed on the whole object then counts one record twice.

Nothing throws. The card catches it before the data does.

Where a free endpoint fits

MonkeyCode is an open-source project offering free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Its operator also advertises a free token allowance on the order of ten million tokens.

Treat those limits as operator claims, not verified facts. I have not independently confirmed them, and you should re-probe rather than assume.

What matters for this workflow is narrower. A free endpoint gives the harness a second card to compare against, without a billing setup first. That is enough to answer the only question this method asks: which end satisfies the contract for this task class today.

Model names, quotas, and retention behavior vary. Pin them in your own card and never hardcode what you have not measured.

Limits you should respect

Three repeats is a smoke test, not a confidence interval. Five latency samples bound nothing. Raise both counts before you route money through them.

The probe prompts are not your traffic. A synthetic needle at 75 percent of filler does not prove behavior on your real documents.

The probes cost tokens. A four-thousand-token filler prompt is not free, even on a free tier.

Endpoints change without notice. A three-day expiry is a guess, not a schedule. Shorten it after any incident.

The harness measures behavior, not cause. It cannot tell you why a field vanished.

A capability card never changes your data policy. If bytes must not leave the machine, do not build the remote path at all.

Who should not use this

Skip this if you owe anyone a formal SLA, or if you cannot re-run the probe when an endpoint changes. Skip it if your data cannot leave the machine, since the remote branch is then pure risk.

Skip it if you need statistically solid latency numbers. A handful of calls will mislead you more than no calls.

Close the loop

Probe both ends with the same harness. Date every measurement. Let the router obey the date, and let it fail closed.

If you want a second endpoint to probe and you would rather not put a card on file first, a free tier is the cheapest way to get one.