Measure the Endpoint Before You Measure the Model

A benchmark score collected through a hosted endpoint measures three things at once, and only one of them is the model. The harness contributes its prompts, its decoding parameters, and its timeouts, while the endpoint contributes routing, truncation, and queueing behaviour that nobody wrote down. When either side drifts, the reported number drifts with it, and the drift is usually invisible because it never appears in the output table. A short conformance suite run before the evaluation separates the three contributors, and it costs a few dozen requests rather than a full evaluation budget.

This article describes such a suite as a reproducible artifact: a nine-line prompt set with a checksum, four probes, a metric definition for each probe, and a reporting rule that decides whether a score may be published at all. The suite is deliberately boring, because its job is to remove ambiguity rather than to produce an impressive chart. Everything below runs against any OpenAI-compatible chat endpoint, which keeps the method portable and the conclusions falsifiable by a reader who reruns it.

The failures a score silently absorbs

The first failure is sampling instability. A hosted endpoint may apply a default temperature, a top-p cut, or a seed policy that differs from the one in the request, and a greedy decode on one host can become a sampled decode on another. The second failure is context truncation, where long inputs are quietly cut at a boundary the caller never sees, so a retrieval task turns into a guessing task. The third is tool-call schema drift, in which the model returns arguments that are almost valid JSON, or a tool call wrapped in prose, and the harness parses it leniently enough to hide the damage. The fourth is silent routing, where the response carries a different model identifier than the one requested, which makes the score unattributable even when every individual answer looks reasonable.

A pinned probe set, not a pile of prompts

The suite reads a JSONL file of probe prompts and records its SHA-256 before the run, so a later reader can tell whether the suite or the endpoint changed. Three probes are enough to start: a repeat probe that sends the same deterministic prompt several times, a needle probe that hides a unique canary string near the end of a long synthetic document, and a schema probe that forces a tool call with a required enum field. The dataset is intentionally tiny and public, because the suite is testing plumbing rather than reasoning quality.

sha256sum probe_set.jsonl
# 6c1f0a...  probe_set.jsonl
python probe_endpoint.py --prompts probe_set.jsonl --out report.json

# probe_endpoint.py — endpoint conformance, not a benchmark
import hashlib, json, os, statistics, time
import requests

BASE = os.environ["MC_BASE_URL"].rstrip("/")   # take this from the provider dashboard
KEY = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL_ID"]

def chat(messages, temperature=0.0, tools=None, timeout=90):
    payload = {"model": MODEL, "messages": messages, "temperature": temperature}
    if tools:
        payload["tools"] = tools
    t0 = time.perf_counter()
    r = requests.post(f"{BASE}/chat/completions",
                      headers={"Authorization": f"Bearer {KEY}"},
                      json=payload, timeout=timeout)
    dt = time.perf_counter() - t0
    r.raise_for_status()
    msg = r.json()["choices"][0]["message"]
    return {"text": msg.get("content") or "",
            "tool_calls": msg.get("tool_calls"),
            "served_model": r.json().get("model"),   # silent routing shows up here
            "latency_s": round(dt, 3)}

def probe_repeat(prompt, repeats=5):
    outs = [chat([{"role": "user", "content": prompt}]) for _ in range(repeats)]
    hashes = {hashlib.sha256(o["text"].encode()).hexdigest()[:12] for o in outs}
    return {"repeats": repeats, "distinct_outputs": len(hashes),
            "agreement": round(1 - (len(hashes) - 1) / repeats, 3),
            "served_models": sorted({o["served_model"] for o in outs if o["served_model"]}),
            "latency_p50": statistics.median(o["latency_s"] for o in outs)}

FILLER = "routine build log line about caching and retries. " * 12

def probe_needle(lines=200):
    needle = "CANARY-7F3A-QQ"
    doc = "\n".join(f"[{i}] {FILLER}" for i in range(lines))
    doc += f"\n[needle] {needle}\n"
    msg = f"Return only the exact string that follows [needle].\n\n{doc}"
    out = chat([{"role": "user", "content": msg}], timeout=180)
    return {"lines": lines, "chars_sent": len(msg), "hit": needle in out["text"]}

TOOL = [{"type": "function", "function": {
    "name": "record_verdict",
    "description": "Record a pass or fail verdict with a short reason.",
    "parameters": {"type": "object", "required": ["verdict", "reason"],
                   "properties": {"verdict": {"type": "string", "enum": ["pass", "fail"]},
                                  "reason": {"type": "string"}}}}}]

def probe_schema():
    out = chat([{"role": "user",
                 "content": "Verdict on the claim 2+2==4. Use the tool."}], tools=TOOL)
    calls = out["tool_calls"] or []
    if not calls:
        return {"schema_valid": False, "why": "no tool call returned"}
    try:
        args = json.loads(calls[0]["function"]["arguments"])
    except json.JSONDecodeError:
        return {"schema_valid": False, "why": "arguments were not JSON"}
    ok = args.get("verdict") in ("pass", "fail") and isinstance(args.get("reason"), str)
    return {"schema_valid": ok, "args": args}

if __name__ == "__main__":
    report = {"prompt_set_sha256": hashlib.sha256(open("probe_set.jsonl", "rb").read()).hexdigest(),
              "repeat": probe_repeat(open("probe_set.jsonl").readline().strip()),
              "needle": probe_needle(),
              "schema": probe_schema()}
    json.dump(report, open(os.environ.get("OUT", "report.json"), "w"), indent=2)
    print(json.dumps(report, indent=2))

Metrics and the controls that make them mean something

The suite reports four numbers and nothing else. Repeat agreement is one minus the fraction of distinct outputs over the repeats, so a perfectly stable endpoint scores 1.0 and an endpoint that returns five different answers scores 0.2. Needle hit rate is binary per run length, and the useful reading is the largest line count that still hits, because that locates the truncation boundary empirically. Schema validity is binary, and it fails loudly on prose-wrapped calls instead of silently repairing them. The fourth metric is the set of served model identifiers, which is a control rather than a score, since any set larger than one invalidates single-model attribution.

Control	Setting	Why it exists
Decoding	temperature 0.0 on every call	removes sampling as an explanation for variance
Concurrency	one request at a time	keeps latency comparable across runs
Prompt identity	SHA-256 of the JSONL file	distinguishes suite edits from endpoint drift
Attribution	record served_model per call	exposes routing that the request did not ask for
Timing	repeat the suite in two separate hours	separates queueing noise from steady-state latency

Why these numbers are not marketing

A marketing figure answers the question of how good the product is, and it is allowed to omit the harness, the retries, and the discarded runs. A conformance report answers a narrower and checkable question: under this exact harness, at this exact hour, with these exact parameters, what did the endpoint return. The narrowness is the point, because a reader who reruns the suite can reproduce the report or contradict it. When a published score omits the conformance report, the reader cannot tell whether a low number came from a weak model or from a truncated prompt, and that ambiguity usually gets resolved in favour of the more flattering story.

A practical reporting rule follows from this. Publish the task score only if repeat agreement is at least 0.8, if the needle probe hits at the task's actual input length, if schema validity is 1.0 for every tool the harness depends on, and if the served model set has exactly one member. If any condition fails, report the conformance result and describe the task score as blocked rather than publishing a number with an asterisk. This rule has cost teams real deadlines, and it has also prevented several comparisons that would not have survived a second run.

Where free access and a free server fit

Running the suite twice, in two separate hours, on a task-sized prompt set is exactly the kind of low-value-but-repetitive work that a paid endpoint discourages. MonkeyCode's free model access and free server option make those repeated runs routine rather than a special event, which is the main reason they appear in this workflow at all; the exact allowance, available models, and server specifications change, so read them from the current dashboard instead of trusting a number quoted in an article. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The limitations matter more than the convenience. The free tier is not a substitute for an SLA, and its latency figures describe a shared queue rather than the model's speed. Nothing here establishes data handling, retention, or residency guarantees, so the probe set should stay public and synthetic and production code should not be pasted into a conformance run until the provider's published policy has been read and accepted. Teams that need contractual uptime, regulated audit trails, or per-run cost accounting with fixed ceilings should run the same suite against an endpoint they already have a contract with, because the method is transport-agnostic and does not depend on any particular provider.

A conformance suite is cheap, unglamorous, and slightly annoying to write the first time. It is also the difference between a benchmark score that survives review and a screenshot that does not, and the free tier is a reasonable place to run it once against a real prompt set before committing an evaluation budget anywhere.