A golden-string match can hide a real regression when tokens, schema validity, or latency move while the visible answer stays the same. Teams that score only pass rate treat a multi-dimensional failure as a single bit, then ship a slower, fatter, or unparseable completion. This article treats those three channels as first-class eval signals with a small HTTP harness. Think of it as a unit test that still asserts equality while the function became quadratic and started printing warnings to stderr.
Most prompt evals inherited the shape of a school exam rather than the shape of a production contract. The grader asks whether the text still looks like the golden answer, then prints a percentage that reads as quality. That percentage is a collapsed histogram: every case is a coin flip, and the envelope around the coin never enters the score. When a model bump rewrites field order, pads a disclaimer, or retries behind the gateway, the string can still contain the expected fragment while parsers, bills, and tail latency quietly move.
Ordinary software already learned this lesson the expensive way, then forgot to copy it into prompt suites. A sort routine can keep returning the same array after someone swaps in a quadratic implementation, and the equality assertion stays green. Continuous integration did not fail because nobody asserted an n-log-n bound, an allocation cap, or a deadline. Language-model evals repeat that mistake whenever they treat the completion as a string and the rest of the response as optional telemetry that belongs in a dashboard nobody opens on merge.
An envelope-aware harness records four channels on every case and fails the run when any channel crosses a pin. Correctness remains a boolean against a golden fragment or a compiled grader you already trust. Schema validity is a second boolean that asks whether the bytes parse and satisfy a tiny contract, even when a human would still call the text right. Size is an integer budget on characters, pinned to the last-good completion rather than to a marketing limit that nobody encoded in git. Time is a wall-clock budget around the HTTP call, which is crude and still enough to catch a path that prints the same city name three times slower.
The decision rule is boring on purpose, because clever weighted scoring is how silent failures sneak back into a green badge. If the golden fragment is missing, the case fails as a correctness regression and the other channels are still reported for diagnosis. If the fragment is present but the payload does not parse, the case fails as a schema regression even though a naive contains() check would pass. If parse and fragment both succeed while characters or milliseconds exceed the pin from the last-good run, the case fails as an envelope regression. Only when all four channels stay inside the pin does the case count as stable, which is a stricter claim than “the answer still mentioned Paris.”
The artifact below is labeled as a runnable proposal, not as a vendor benchmark and not as a production SLO. It talks to any OpenAI-style /v1/chat/completions endpoint, writes a JSON report, and exits nonzero when a channel breaks. Size uses character count divided by four as a blunt token proxy; that proxy misreads CJK, code fences, and special tokens, and the script says so in the report. Wall time includes DNS, TLS, and queueing, so a cold start can look like a model regression unless you keep a slack band and, later, a median of repeats.
#!/usr/bin/env python3
"""envelope_eval.py — proposal: fail when text matches but schema/size/time drift."""
from __future__ import annotations
import argparse, json, os, time, urllib.request
from pathlib import Path
SLACK_SIZE = 1.10 # 10% character slack vs last-good pin
SLACK_TIME = 1.15 # 15% wall-time slack vs last-good pin
def chat(url: str, model: str, prompt: str, timeout: float) -> tuple[str, float]:
body = json.dumps({
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
}).encode()
req = urllib.request.Request(
url, data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {os.environ.get('EVAL_TOKEN', '')}"},
)
t0 = time.perf_counter()
with urllib.request.urlopen(req, timeout=timeout) as resp:
payload = json.loads(resp.read().decode())
ms = (time.perf_counter() - t0) * 1000
text = payload["choices"][0]["message"]["content"]
return text, ms
def schema_ok(text: str, spec: dict | None) -> bool:
if not spec:
return True
try:
obj = json.loads(text)
except json.JSONDecodeError:
return False
if spec.get("type") == "object" and not isinstance(obj, dict):
return False
for key in spec.get("required", []):
if key not in obj:
return False
return True
def eval_case(case: dict, pins: dict, url: str, model: str) -> dict:
text, ms = chat(url, model, case["prompt"], timeout=case.get("timeout_s", 30))
chars = len(text)
est_tokens = max(1, chars // 4)
correct = case["expect_fragment"] in text
schema = schema_ok(text, case.get("schema"))
pin = pins.get(case["id"], {})
size_fail = bool(pin) and chars > pin["chars"] * SLACK_SIZE
time_fail = bool(pin) and ms > pin["ms"] * SLACK_TIME
if not correct:
channel = "correctness"
elif not schema:
channel = "schema"
elif size_fail:
channel = "size"
elif time_fail:
channel = "time"
else:
channel = "stable"
return {
"id": case["id"], "channel": channel, "correct": correct,
"schema": schema, "chars": chars, "est_tokens": est_tokens,
"ms": round(ms, 1), "preview": text[:180],
}
def main() -> int:
p = argparse.ArgumentParser()
p.add_argument("--fixtures", default="fixtures.json")
p.add_argument("--pins", default="last_good.json")
p.add_argument("--write-pins", action="store_true")
p.add_argument("--url", default=os.environ.get("EVAL_URL", "http://127.0.0.1:8080/v1/chat/completions"))
p.add_argument("--model", default=os.environ.get("EVAL_MODEL", "local"))
args = p.parse_args()
cases = json.loads(Path(args.fixtures).read_text())
pins = json.loads(Path(args.pins).read_text()) if Path(args.pins).exists() else {}
rows = [eval_case(c, pins, args.url, args.model) for c in cases]
Path("eval_report.json").write_text(json.dumps(rows, indent=2))
if args.write_pins:
new_pins = {r["id"]: {"chars": r["chars"], "ms": r["ms"]} for r in rows}
Path(args.pins).write_text(json.dumps(new_pins, indent=2))
print("wrote last_good pins; review before committing")
return 0
broken = [r for r in rows if r["channel"] != "stable"]
for r in rows:
print(f"{r['id']:16} {r['channel']:12} chars={r['chars']:<5} ~tok={r['est_tokens']:<5} {r['ms']}ms")
print(f"{len(rows) - len(broken)}/{len(rows)} stable")
return 1 if broken else 0
if __name__ == "__main__":
raise SystemExit(main())
Pair that runner with a fixture file that stores the contract beside the prompt, because a side spreadsheet will not get reviewed in the same diff. The schema is intentionally small: required keys only, no attempt to reimplement a full JSON Schema library inside the eval. Last-good pins are a second file so you can accept a slower but legitimate prompt change without editing the golden fragment. The first run should use --write-pins under human review; later runs should refuse to rewrite pins from CI, or a noisy afternoon will launder a regression into the baseline.
[
{
"id": "capital_json",
"prompt": "Reply with JSON only: {\"city\": string, \"country\": string} for France's capital.",
"expect_fragment": "Paris",
"schema": {"type": "object", "required": ["city", "country"]},
"timeout_s": 20
},
{
"id": "refuses_tools",
"prompt": "User asked for weather. Reply with JSON {\"error\": \"missing_location\"} and nothing else.",
"expect_fragment": "missing_location",
"schema": {"type": "object", "required": ["error"]},
"timeout_s": 20
}
]
python3 envelope_eval.py --write-pins # review last_good.json, then commit
EVAL_URL=http://127.0.0.1:8080/v1/chat/completions \
EVAL_MODEL=local \
python3 envelope_eval.py; echo exit:$?
python3 -c "import json; rows=json.load(open('eval_report.json'));\
print({r['channel']: sum(1 for x in rows if x['channel']==r['channel']) for r in rows})"
Slack of about fifteen percent on time and ten percent on size absorbs handshake noise without hiding a doubling, which is the failure mode this harness exists to name. If you already calibrate a noise floor for grader disagreement across samples, keep that work in the correctness channel; this script answers a different question. The question is whether a still-correct answer has become expensive, ill-formed, or slow enough that the application will time out even though the substring survived. Mixing those questions into one weighted score is how a schema break gets averaged away by two easy factual cases.
Where this loop tends to die in practice is not the math. It is the cost of actually running extra samples every time a system preamble gains an adjective. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to host this script beside an OpenAI-compatible endpoint so the envelope pins get regenerated instead of rotting in a gist. That is an operational convenience, not a claim about named models, quotas, hardware, or how long the free path lasts, and those details should be read from the product itself before anyone schedules a fleet.
Do not treat a free path as a capacity guarantee or a latency SLA, because unspecified limits are not pins you can commit. Keep customer text out of golden prompts, hash the fixture directory, and fail the job if the fixture hash changes without a reviewed pin update. If several people share a server, freeze the prompt file in the same commit as last_good.json, or a neighbor's experiment will rewrite your baseline while your pass rate stays 100. Secrets belong in EVAL_TOKEN, not in the fixture that will be copied into a gist during the next incident review.
This approach is the wrong tool when the product already has a real APM budget, load tests, and a parser sitting in front of the model. Those systems measure time and schema on live traffic, which is a better envelope than a dozen JSON fixtures living in git. It is also the wrong tool for embeddings, ranking, or multimodal outputs, because a character budget does not mean anything for a vector or an image. Teams that need human preference data will not obtain it from schema checks and a stopwatch, and they should not pretend a green envelope report is a substitute for raters.
A second limitation is statistical, and it is the one that makes envelope evals look flaky in the first week. One timed HTTP call per case will flap on a cold start, a rate limit, or a noisy neighbor, and the slack band only hides small flaps. If you need confidence intervals, run repeats and fail on median and p95; the script only sketches that design with a single sample so the control flow stays readable. Schema checks that accept any object with two keys will still miss a swapped field that the application later interprets incorrectly, so keep a real contract and a downstream parser test, not an empty {}.
The useful outcome is a report that names the channel, not a single red X that sends people back into the raw completion log. Correctness failures still matter, and they should remain the first column because a missing city is not an envelope story. Envelope failures are the ones that used to ship because the demo answer looked unchanged in the pull request while the payload doubled and the client started retrying. Once the last-good pins live beside the fixtures, a model bump, a prompt adjective, or a new system preamble has to justify extra bytes and extra time, not only preserve a substring that a contains() check can still see.
If you already keep graders as code, add the envelope pins to the same process rather than standing up a second religion with a different report format. The cheapest honest loop is the one that runs on every prompt diff with budgets a reviewer can explain in a comment. A scheduled run against free model access is enough to keep those pins from rotting, and that is the entire operational claim this article is willing to make.
Top comments (1)
Strong point. Treating correctness as only a string match can hide exactly the kinds of regressions that hurt in production—schema drift, token growth, and latency. I especially like the idea of keeping last-good envelope pins beside the fixtures. In larger agent systems, though, latency can vary a lot because of tool calls and retries. Have you considered separating model latency from orchestration/tool latency in the same evaluation report?