Route by Evidence, Not Vibes: A 90-Minute Lab for Splitting Work Between a Free Model Endpoint and a Free Server

The most useful benchmark you can run this quarter is not a leaderboard score; it is a routing table your own team can defend with measurements. Public comparisons answer which model wins on someone else's prompts, while your real decision is narrower: which execution path satisfies the latency, reliability, and token envelope for each class of task you ship. This lab turns that decision into a 90-minute workshop with a fixed task set, a JSONL evidence log, and a policy table students can rerun after every endpoint change.

The framing matters because benchmarking season produces a lot of impressive aggregate numbers and very little per-task routing advice. A model that tops a public chart can still be the wrong choice for a 40-token extraction call that needs a 300 ms p95. Conversely, a modest endpoint becomes the correct default once you know its failure rate under your own retry policy.

The artifact: a 12-row routing ledger

Students leave with one file per execution path plus one derived table. The ledger has twelve rows, meaning three task kinds times four paths, and each row carries the same columns so comparisons stay honest.

Column	Type	Why it exists
task_id	string	Ties every run back to the frozen prompt set
kind	enum	Groups extraction, summarization, code-edit tasks
path	enum	remote-endpoint, self-hosted, hybrid, cached
ok	bool	Records schema-valid output, not just HTTP 200
latency_ms	float	Wall clock from request to parsed response
prompt_tokens / completion_tokens	int | null	Null when the endpoint omits usage
error_class	string | null	Timeout, 429, 5xx, schema-drift, empty

Two conventions keep the numbers comparable across the room. First, freeze the prompt set before anyone starts measuring, because a prompt edited mid-lab invalidates every earlier row. Second, count schema-valid responses as the only successes, since a fast malformed answer is a failure that happens to look cheap.

90-minute workshop outline

Minutes	Segment	Output
0-10	Frame the routing question	One sentence per task kind describing the envelope
10-20	Freeze prompts, write the validator	tasks.jsonl plus validate.py
20-45	Exercise 1: measure the hosted endpoint	runs/endpoint.jsonl
45-60	Exercise 2: measure the self-hosted path	runs/server.jsonl
60-75	Exercise 3: apply the policy table	routing.md
75-90	Exercise 4: rerun and debrief	Diff report between two runs

Exercise 1: instrument one path

The harness below is a reference implementation for the lab, not a verified client library. Adapt the request body and the response parser to the endpoint you actually use, and treat any token counts as unknown when the response omits a usage block.

#!/usr/bin/env python3
# route_probe.py - reference harness for the routing lab.
# Not executed against a live endpoint in this draft; adapt path and parser.
import argparse, json, os, time, urllib.request, urllib.error

TASKS = [
    {'id': 'extract-01', 'kind': 'extraction', 'prompt': 'Return JSON with keys id, total, currency from: <invoice text>'},
    {'id': 'summ-01',    'kind': 'summarization', 'prompt': 'Summarize in 3 bullets for an on-call engineer: <incident log>'},
    {'id': 'edit-01',    'kind': 'code-edit', 'prompt': 'Rewrite this function to return early on empty input: <code>'},
]

def call(base_url, api_key, path, prompt, timeout):
    payload = json.dumps({'messages': [{'role': 'user', 'content': prompt}]}).encode()
    req = urllib.request.Request(
        base_url.rstrip('/') + path,
        data=payload,
        headers={'Content-Type': 'application/json', 'Authorization': 'Bearer ' + api_key},
    )
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            raw = resp.read().decode('utf-8', 'replace')
        return time.perf_counter() - start, raw, None
    except urllib.error.HTTPError as exc:
        return time.perf_counter() - start, '', 'http_' + str(exc.code)
    except Exception as exc:
        return time.perf_counter() - start, '', type(exc).__name__

def usage_of(raw):
    try:
        blob = json.loads(raw)
    except ValueError:
        return None, None
    usage = blob.get('usage') or {}
    return usage.get('prompt_tokens'), usage.get('completion_tokens')

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--base-url', default=os.environ.get('PROBE_BASE_URL', ''))
    ap.add_argument('--api-key', default=os.environ.get('PROBE_API_KEY', ''))
    ap.add_argument('--path', default='/v1/chat/completions')
    ap.add_argument('--label', required=True)
    ap.add_argument('--timeout', type=float, default=30.0)
    ap.add_argument('--out', required=True)
    args = ap.parse_args()
    with open(args.out, 'a', encoding='utf-8') as sink:
        for task in TASKS:
            latency, raw, err = call(args.base_url, args.api_key, args.path, task['prompt'], args.timeout)
            ptok, ctok = usage_of(raw)
            row = {'task_id': task['id'], 'kind': task['kind'], 'path': args.label,
                   'ok': bool(raw) and err is None, 'latency_ms': round(latency * 1000, 1),
                   'prompt_tokens': ptok, 'completion_tokens': ctok, 'error_class': err}
            sink.write(json.dumps(row) + '\n')
            print(json.dumps(row))

if __name__ == '__main__':
    main()

Run it once per path with an explicit label, and keep the label stable so later diffs stay readable.

PROBE_BASE_URL=$ENDPOINT_URL PROBE_API_KEY=$ENDPOINT_KEY \
  python route_probe.py --label remote-endpoint --out runs/endpoint.jsonl

PROBE_BASE_URL=http://127.0.0.1:8080 PROBE_API_KEY=local \
  python route_probe.py --label self-hosted --out runs/server.jsonl

Exercise 2: replace raw latency with a per-task verdict

A single p50 number hides the only figure that changes routing decisions, which is the share of schema-valid responses inside your latency budget.

# policy.py - turn JSONL rows into a routing recommendation
import json, sys
from collections import defaultdict

BUDGET_MS = {'extraction': 800, 'summarization': 4000, 'code-edit': 6000}
rows = [json.loads(line) for line in open(sys.argv[1], encoding='utf-8')]
buckets = defaultdict(list)
for row in rows:
    buckets[(row['kind'], row['path'])].append(row)
for (kind, path), group in sorted(buckets.items()):
    good = [r for r in group if r['ok'] and r['latency_ms'] <= BUDGET_MS[kind]]
    hit = len(good) / len(group)
    print(kind, path, 'n=' + str(len(group)), 'in_budget=' + format(hit, '.2f'))

Exercise 3: apply the policy table

The thresholds below are starter values for the workshop, not measured findings; each team rewrites them from its own product requirements before writing the final routing document.

Observed in-budget rate	Chosen route	Escalation rule
>= 0.95	Hosted endpoint only	Fail over on two consecutive errors
0.80-0.95	Hosted primary, server fallback	Fallback after one timeout over budget
0.50-0.80	Split by task kind	Route by the ledger, not by default
< 0.50	Server preferred, endpoint for spikes	Endpoint capped by a token budget

Exercise 4: prove the rerun is boring

Students rerun both paths after lunch and diff the two ledgers. The deliverable is not a nicer chart; it is a one-paragraph change note explaining why any verdict moved, which is the habit that survives contact with a real incident.

Worked example (synthetic, for teaching only)

The numbers below are invented placeholders that show the arithmetic, not results from any provider. Extraction scores 0.97 in budget on the hosted endpoint and 0.64 on the self-hosted path, so extraction stays remote. Summarization scores 0.88 hosted and 0.90 self-hosted, so it moves to the server with an endpoint fallback. Code-edit scores 0.55 hosted and 0.52 self-hosted, which usually means the envelope is wrong rather than the model: raise the budget or split the task before picking a winner. Notice that no row required a leaderboard lookup.

Where a free tier fits the lab

MonkeyCode (open source per its maintainers) is a reasonable place to run this lab, because the operator describes free model access and a free server option, with the current allowance stated as 10 million free tokens. I have not independently verified those terms, so confirm the present quota and the conditions attached to the server option in the project documentation before you size a class around them, and assume nothing about model names or hardware.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to teach this lab before requesting paid capacity, start the endpoint path on the free allowance and run the server path on the free server option; that combination covers both exercises without a procurement conversation.

Limitations and who should not use this

• Teams with a strict data-residency or compliance review should settle that review before measuring anything, since the routing table is worthless if half the routes are disallowed.
• Anyone who needs reproducible absolute latency numbers should not trust a shared free tier, because noisy neighbours make wall-clock comparisons across days unreliable.
• Small task sets overfit badly; twelve ledger rows justify a conversation, not a company-wide migration.
• Groups without a response validator should stop and build one first, because every downstream number depends on what counts as success.

The rerun kit

Ship four files with the workshop: tasks.jsonl (frozen prompts), route_probe.py, policy.py, and routing.md (the filled table plus one paragraph of reasoning). Add a runs/ directory and a one-line make rerun target so the next cohort reproduces the same comparison instead of retyping commands from memory.