A free model scores 30% on your eval. The obvious conclusion: the model is weak. Swap it, and the score barely moves. That is the moment to stop treating the number as a verdict and start treating it as a symptom.
An eval score is a stack. Model, prompt, harness, ground truth, and environment all contribute. Any layer can drag the number down, and free tiers add two more variables: shared endpoints and rate limits. This article is a short glossary, a decision tree for isolating the failing layer, and a 30-line probe you can run on a free server.
Glossary: the five layers
- Eval set — the input/output pairs you score against. Garbage here makes every other layer look broken.
- Harness — the code that sends prompts, parses outputs, and compares them to ground truth. The most common source of phantom failures.
- Ground truth — the expected output you trust. It can be wrong, ambiguous, or silently outdated.
- Determinism — whether the same prompt produces the same output. At temperature 0, most models are near-deterministic; on shared free endpoints, they are not always.
-
Context truncation — when the input exceeds the model's window and the relevant part never reaches the model. The API usually reports this as
finish_reason: "length".
The decision tree
Run the failing item alone, twice, before changing anything.
Step 1 — Reproduce. Same input, different outputs? → Leaf 1: variance. Same output? → Step 2.
Step 2 — Read the raw output, not the score. Output is valid but the harness marked it wrong? → Leaf 2: harness bug. Output is actually wrong? → Step 3.
Step 3 — Check the context. The prompt was truncated, or generation stopped at the length limit? → Leaf 3: context. Full context and the output is still wrong? → Step 4.
Step 4 — Audit the ground truth. Ground truth is wrong or ambiguous? → Leaf 4: bad label. Ground truth is right? → Leaf 5: the model.
Worked leaves
The numbers below are illustrative; the point is the branch, not the exact figures.
Leaf 1 — variance. A 20-item eval scored 55% on run one and 70% on run two. Five items flipped between runs. The fix was boring: temperature 0, three runs, report the median. The score stabilized at 65%. On a free endpoint, variance is often provider load, not the model.
Leaf 2 — harness bug. The model returned valid JSON wrapped in
```json fences. The parser expected raw JSON. Eight of twenty items "failed" on parse before the model was ever judged. The fix is a tolerant extractor: strip code fences, then parse. This is the most common leaf and the cheapest to fix.
Leaf 3 — context. An eval item included a 40k-token source file. The relevant function lived at the end of the file. The model answered from the first 8k tokens and missed it. finish_reason was length. The fix: shrink the item or move the relevant symbol to the top.
Leaf 4 — bad label. Ground truth said a function should return None. The spec said it should raise ValueError. The model raised. The label was wrong. Fixing one label moved the score by ten points and changed which model looked better.
Leaf 5 — the model. All checks pass. The output is stable, parsed correctly, fully in context, and the ground truth is defensible. The model simply cannot do the task. This is the only leaf where swapping models is the right move — and now you have evidence, not a hunch.
A 30-line probe
The probe below checks the two cheapest failure layers — variance and truncation — before you touch the harness or the data. It needs only a chat-completions endpoint.
python
# eval_probe.py — separate model signal from harness noise.
# Usage:
# export OPENAI_BASE_URL=... OPENAI_API_KEY=... MODEL=...
# python eval_probe.py "Return the JSON {\"ok\": true}."
import os
import sys
from openai import OpenAI
PROMPT = sys.argv[1] if len(sys.argv) > 1 else "Return the JSON {\"ok\": true}."
RUNS = int(os.environ.get("RUNS", "3"))
client = OpenAI(
base_url=os.environ["OPENAI_BASE_URL"],
api_key=os.environ["OPENAI_API_KEY"],
)
outputs, reasons = [], []
for i in range(RUNS):
resp = client.chat.completions.create(
model=os.environ["MODEL"],
messages=[{"role": "user", "content": PROMPT}],
temperature=0,
)
outputs.append(resp.choices[0].message.content)
reasons.append(resp.choices[0].finish_reason)
print(f"run {i + 1}: finish_reason={reasons[-1]} len={len(outputs[-1] or '')}")
unique = {o for o in outputs if o is not None}
print(f"\ndeterminism: {len(unique)} unique output(s) across {RUNS} runs")
if len(unique) > 1:
print("verdict: variance — fix temperature and rerun before trusting the score")
elif any(r == "length" for r in reasons):
print("verdict: truncation — the context window cut the generation; shrink the item")
else:
print("verdict: stable — the score reflects model + prompt + data, not noise")
The workflow above runs entirely on free resources: the model access bundled with MonkeyCode, an open-source project that currently offers a 10 million token allowance and a free server option. The probe itself is provider-agnostic — point it at any chat-completions endpoint, free or paid.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Limitations
This tree assumes the failure is reproducible. Rare flakes need more runs, not a decision tree. The probe checks variance and truncation, not semantic correctness — you still need ground truth or a judge model for that. And this is triage, not a replacement for a proper eval framework. If you are tracking a model across versions, build a regression suite after you know which layer was failing.
Who should not use this: teams with paid, high-stakes evals that need statistical rigor — use a real framework with confidence intervals — and anyone evaluating long-context reasoning, where truncation needs a different detection method. Free-tier terms also change. The 10 million allowance and the free server are what MonkeyCode offers as of this writing; verify the current terms before building a pipeline on them.
The takeaway
A low score is a starting point, not a verdict. Run the tree, find the leaf, fix the cheapest layer first. Most eval failures are not the model. When it is the model, you will know — because you checked everything else.
The whole check fits inside a free token allowance, so it costs nothing to run. If you adapt the probe to your own evals, I would be curious which leaf your failures land on.
Top comments (0)