Measure a Free Model Route With a Golden Set, Not a Leaderboard

The first number I track when evaluating a free model route is not the public benchmark delta. It is the semantic failure count in a 100-case golden set. A public benchmark can be flat while your application's specific tool calls, JSON schemas, and failure states move.

I start with a golden set: 100–300 request/response pairs captured from the current route. Each case stores the expected tool call and a small list of acceptable result patterns. The set must include ambiguous prompts, permission denials, empty-result cases, and at least one request that should deliberately fail. If a candidate cannot reproduce that failure, it is not a drop-in route.

The evaluator is deliberately boring. It sends each case through the baseline and candidate adapters, records latency, and compares structured outcomes instead of whole prose.

import json
import time
from dataclasses import dataclass
from pathlib import Path

@dataclass
class Case:
    id: str
    prompt: str
    expected_tool: str
    allowed_result_patterns: list[str]

def evaluate(cases, run):
    failures = []
    for case in cases:
        start = time.perf_counter()
        result = run(case.prompt)
        latency_ms = (time.perf_counter() - start) * 1000
        if result.tool != case.expected_tool:
            failures.append((case.id, "tool_mismatch", result.tool, latency_ms))
            continue
        if not any(pattern in result.text for pattern in case.allowed_result_patterns):
            failures.append((case.id, "assertion_miss", result.text[:120], latency_ms))
    return failures

def load_cases(path: Path):
    return [Case(**item) for item in json.loads(path.read_text())]

# run_baseline and run_candidate are thin HTTP adapters over each provider.
for name, run in {"baseline": run_baseline, "candidate": run_candidate}.items():
    failures = evaluate(load_cases(Path("golden_set.json")), run)
    print(json.dumps({"route": name, "failures": len(failures)}))

The comparison is not between prose quality scores. It is between the set of semantic failures each route produces. I use three signals before changing routing:

Signal	Leaderboard	Golden set	Shadow route
Quality view	aggregate tasks	your exact requests	your exact requests plus live traffic
Cost view	provider list price	not measured	actual tokens in your mix
Failure visibility	low	exact mismatch	production error and response code
Setup cost	none	medium	high

To keep the evaluation from depending on my laptop or a paid inference bill, I used MonkeyCode's free model access for the candidate route and its free server option to host the evaluator. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness itself treats the provider as an HTTP seam, so the same script can evaluate whichever model your deployment already uses.

A decision table is more useful than a leaderboard position:

Evidence	Action
Candidate failure rate ≤ baseline and p95 latency within budget	allow shadow writes
Tool mismatch rate above 5%	keep candidate read-only behind a feature flag
Token cost drops by 30% but assertion misses rise	reopen the system prompt and retry before routing
Candidate cannot reproduce the deliberate failure cases	stop the rollout; do not widen permissions

The limitations are real. A golden set drifts as prompts and schemas change, so it must be rebuilt from traces on a schedule. It measures behavioral compatibility, not capability, security, or throughput under load. Free tiers can have cold starts, quotas, or availability limits that invalidate latency numbers, and a route that passes locally can still fail when permissions cross a production boundary. If your API is still in a single provider call, add the provider seam first; this evaluator will not help until the route is an explicit contract.

What usually fails in your stack is not the model choice but the handoff at the route boundary. Which layer in your current setup is least stable: prompt, parse, permission check, or provider timeout? I would be interested in a concrete failure response code you saw when a cheaper model was swapped in.