I Counted Drops as Wrongs. The Chart Was Theater.

The first number on an eval dashboard is usually a lie. Not a scam. A folding error. You asked a model for an answer, the path blinked, and your scorer filed the blink under incorrect.

I stopped scoring completions until I scored the envelope.

If you cannot tell a drop from a wrong, you are not ranking models. You are ranking weather. Shared free endpoints make the weather louder: idle processes, truncated streams, empty choices that still ride in on HTTP 200. The status looks fine. The grade looks like failure. That is how a leaderboard turns into theater.

I wanted a harness that fails loudly at the transport layer before it whispers at the semantics layer. So I planted six failure modes, ran a classifier over twenty-four envelopes, and watched a naive pass-rate collapse on purpose. The code below is the experiment. The percentages are the fixture talking, not a vendor scoreboard.

The folding error

Most agent evals I see still do some version of success = (http == 200 and expected in text). Cute. Also how you launder a timeout into a dumb model.

Think of it like grading a take-home by weighing the envelope. A missing envelope is not a wrong proof. A torn envelope is not a wrong proof. A proof written in the wrong schema is closer, but it is still not the same as a proof that multiplies instead of factors. Mix them and you will fire the student who had a postal strike.

Free inference is the postal strike. I do not mean that as an insult. I mean that a path you are not paying for will drop, stall, and truncate in ways a dedicated box will not, and your scorer has to admit that or it will keep "proving" that cheaper models are worse at arithmetic when they were worse at staying on the line.

That is why I care about a free model path and a free server option at all. Not because I want an adjective. Because I want a hostile mailbox. If the harness survives a mailbox, it will not panic on a quiet one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free model access and free server option are the mailbox I have in mind when I say "hostile." I am not naming models, quoting quotas, or claiming uptime. If you cannot reproduce a claim from the fixture in this file, treat the claim as unset.

Six envelopes, one bit

I keep six labels, and I refuse to let them share a boolean. drop is a refused connection, a reset, a client timeout, a 429, a 503. empty is a 200 with no choices or a null content string. truncated is a finish_reason of length, or JSON that dies mid-key. schema is parseable JSON that misses a required field. wrong is valid JSON with the wrong answer. right is the boring one.

Ask yourself what your current CSV does with empty. If the answer is "fail," you already know why last week's ranking felt jumpy. The model did not change. The path coughed, and you graded the cough.

I did not need a live fleet to test the scorer. I needed lies I controlled. Twenty-four envelopes, four of each label, stuffed into memory. The naive scorer treats anything that is not right as a model miss. The envelope scorer reports two numbers: yield, meaning bodies that were even gradeable, and accuracy-on-yield, meaning rights over rights plus wrongs.

Here is the whole trick. Yield answers "did the path hand me an object?" Accuracy-on-yield answers "was the object correct?" If you publish only the product of those, you are back to theater.

The planted run

Save this as envelope_eval.py. It never calls a network. That is the point. If a scorer cannot recover a lie you planted on disk, it will not recover a lie a free server plants for you at 4 p.m.

#!/usr/bin/env python3
"""Planted-envelope scorer. Run: python envelope_eval.py"""
from __future__ import annotations

import json
from collections import Counter
from dataclasses import dataclass
from typing import Any, Optional

@dataclass
class Envelope:
    http_status: int
    elapsed_ms: float
    finish_reason: Optional[str]
    body: Any
    expected: dict

def classify(env: Envelope) -> str:
    if env.http_status in {0, 408, 429, 500, 502, 503, 504}:
        return "drop"
    if env.http_status != 200:
        return "drop"
    if not isinstance(env.body, dict):
        return "empty"
    choices = env.body.get("choices") or []
    if not choices:
        return "empty"
    content = (choices[0].get("message") or {}).get("content")
    reason = env.finish_reason or choices[0].get("finish_reason")
    if content in (None, ""):
        return "empty"
    if reason == "length":
        return "truncated"
    try:
        parsed = json.loads(content) if isinstance(content, str) else content
    except json.JSONDecodeError:
        return "truncated"
    if not isinstance(parsed, dict) or "answer" not in parsed:
        return "schema"
    if parsed.get("answer") != env.expected.get("answer"):
        return "wrong"
    return "right"

def plant() -> list[Envelope]:
    expected = {"answer": 42}
    good = {"choices": [{"message": {"content": '{"answer": 42}'}, "finish_reason": "stop"}]}
    bad = {"choices": [{"message": {"content": '{"answer": 7}'}, "finish_reason": "stop"}]}
    missing = {"choices": [{"message": {"content": '{"value": 42}'}, "finish_reason": "stop"}]}
    cut = {"choices": [{"message": {"content": '{"answ'}, "finish_reason": "length"}]}
    empty = {"choices": []}
    rows: list[Envelope] = []
    for _ in range(4):
        rows.append(Envelope(0, 30000, None, None, expected))
        rows.append(Envelope(200, 800, "stop", empty, expected))
        rows.append(Envelope(200, 1200, "length", cut, expected))
        rows.append(Envelope(200, 900, "stop", missing, expected))
        rows.append(Envelope(200, 700, "stop", bad, expected))
        rows.append(Envelope(200, 650, "stop", good, expected))
    return rows

def main() -> None:
    labels = [classify(env) for env in plant()]
    counts = Counter(labels)
    n = len(labels)
    naive = sum(lab == "right" for lab in labels) / n
    gradeable = counts["wrong"] + counts["right"]
    acc = counts["right"] / gradeable if gradeable else float("nan")
    print("counts:", dict(counts))
    print(f"naive_pass_rate={naive:.3f}")
    print(f"yield={gradeable / n:.3f}")
    print(f"accuracy_on_yield={acc:.3f}")

if __name__ == "__main__":
    main()

Run it.

python envelope_eval.py

You should get four of each label, naive_pass_rate=0.167, yield=0.333, accuracy_on_yield=0.500. That naive 16.7% is true and useless. It says the system is bad at the task. It is not. The system is 50% accurate on the eight gradeable semantic answers, and sixteen of twenty-four envelopes never earned a semantic grade. If I had shipped the naive number, I would have "proven" a model regression that was actually four drops, four empties, four truncations, and four schema misses I planted myself.

Would you publish 16.7% in a weekly review? I have. That is the part that still bothers me.

What I do with each label

I keep a small matrix next to the scorer so I cannot "just retry it" at 11 p.m. and call that science.

Label	Counts as model evidence?	Action
drop	no	retry with a budget, then park the row
empty	no	retry once, then park
truncated	no	raise max tokens or repair, cap the repairs
schema	no	repair or fail the harness, not the model
wrong	yes	score it, do not retry
right	yes	score it, do not retry

Retry only drops and empties. Repair only truncated and schema, and cap the repairs so you do not invent a second model with your JSON fixer. Score only wrong versus right. Never retry a wrong. Wrong is information. A retry on a wrong is how you accidentally bake temperature into a pass@1 and then argue with your own chart.

Where a free path helps, and where it breaks

I use a free model and a free server as a canary for the harness, not as a verdict on the product under test. The canary's job is to produce ugly envelopes on a weekday afternoon. If my classifier labels them, the paid run gets a cleaner CSV. If my classifier folds them, I am about to buy myself a fake regression.

It breaks in the obvious ways. A free path is not a latency SLA. It is not a promise that temperature 0 is deterministic. It is not a substitute for a held-out human grade. If your paper needs matched seeds across labs, do not use a shared free box as the official judge. If you are on a deadline and you cannot afford a thirty-second stall before a drop, this is the wrong mailbox. If you need to claim "the model scored 71%," this article will not give you that sentence, and neither should a canary.

I also do not use the same endpoint for candidate and judge when I can avoid it. Shared load couples their clocks. The judge gets slow, your client times out, and you file a drop as a disagreement. That loop is how evals invent opinions. Two mailboxes. One grade.

What I log now

Every row gets http_status, latency_ms, finish_reason, bytes, label, and only then answer. The live probe looks like this, and I keep --max-time boringly small so the client timeout is a fact instead of a hang.

curl -sS -D /tmp/hdr.txt --max-time 30 \
  -H "content-type: application/json" \
  --data @prompt.json \
  "$ENDPOINT/v1/chat/completions" | tee /tmp/body.json | \
  python -c "import json,sys; d=json.load(sys.stdin); c=d.get('choices') or []; print({'n_choices': len(c), 'finish': c[0].get('finish_reason') if c else None, 'bytes': len(json.dumps(d))})"
wc -c /tmp/body.json

If n_choices is 0, I do not regex the content for the golden string. There is no content. Stop pretending. HTTP 200 is a door. It is not a grade.

Who should ignore this? If you already isolate transport in your eval warehouse, you do not need my JSONL. If you are demoing a toy prompt in a notebook, you do not need six labels. If you are trying to make a free server look like a dedicated GPU, you are solving a press-release problem, and this workflow will not help you.

I still run the planted file before I point the same scorer at anything with a bill, or anything without one. Steal the classifier first. If you want a mailbox that will actually drop on you while you do it, MonkeyCode's free model access and free server option are one place to aim the canary. Read the labels. Do not publish the folded number.