Run a Shadow Contract Against a Free Model Route Before You Merge the Switch

Run a Shadow Contract Against a Free Model Route Before You Merge the Switch

Your primary LLM route works. You find a free model route with a 30 million token allowance and a free server option, so you call it directly from one feature branch. The demo returns a plausible answer and the PR goes green. Later the same branch fails in CI with a 429, a timeout, or an empty choices array that your code assumed would never be empty.

The failure is not model quality. It is that the free route became a dependency before it was measured as a contract.

MonkeyCode is one vendor with a public free model route. Its product materials describe an open-source project, a 30 million token allowance, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating the 30 million token figure as an operator-supplied input, not as an independent guarantee, so the harness below records what the endpoint actually returns.

The shadow contract

Keep the free route in a shadow lane. Primary traffic does not change. The shadow route receives the same prompt cases, but a merge is blocked only when it violates a response contract, not when one answer sounds slightly different.

The contract has six fields per request: HTTP status, latency, output text, prompt tokens, completion tokens, and error text. That is more useful than comparing two paragraphs by eye.

Create a small Python client that sends identical cases to both routes and records the response envelope.

import argparse
import json
import os
import time
from dataclasses import asdict, dataclass
from pathlib import Path

import httpx

@dataclass
class RouteResult:
    case_id: str
    route: str
    ok: bool
    status_code: int
    latency_ms: int
    output_text: str
    prompt_tokens: int | None = None
    completion_tokens: int | None = None
    error: str | None = None

def call_route(client, base_url, api_key, model, case):
    started = time.perf_counter()
    payload = {
        'model': model,
        'messages': [
            {'role': 'system', 'content': case.get('system', 'You are a precise assistant.')},
            {'role': 'user', 'content': case['user']},
        ],
        'temperature': case.get('temperature', 0),
        'max_tokens': case.get('max_tokens', 160),
    }
    url = base_url.rstrip('/') + '/chat/completions'
    headers = {'Authorization': 'Bearer ' + api_key}
    try:
        resp = client.post(url, headers=headers, json=payload, timeout=case.get('timeout_s', 30))
        latency_ms = int((time.perf_counter() - started) * 1000)
        if resp.status_code != 200:
            return RouteResult(case['id'], 'route', False, resp.status_code, latency_ms, '', error=resp.text[:300])
        body = resp.json()
        choices = body.get('choices') or []
        if not choices:
            return RouteResult(case['id'], 'route', False, 200, latency_ms, '', error='empty choices')
        usage = body.get('usage') or {}
        text = choices[0].get('message', {}).get('content', '')
        return RouteResult(
            case['id'], 'route', True, 200, latency_ms, text,
            usage.get('prompt_tokens'), usage.get('completion_tokens'),
        )
    except Exception as exc:
        latency_ms = int((time.perf_counter() - started) * 1000)
        return RouteResult(case['id'], 'route', False, 0, latency_ms, '', error=repr(exc))

def run(cases_path, routes):
    cases = [json.loads(line) for line in Path(cases_path).read_text().splitlines() if line.strip()]
    report = []
    with httpx.Client() as client:
        for case in cases:
            for label, base, model in routes:
                api_key = os.environ[label.upper() + '_API_KEY']
                result = call_route(client, base, api_key, model, case)
                result.route = label
                report.append(asdict(result))
                print(json.dumps({**asdict(result), 'user': case['user'][:80]}))
    Path('shadow_report.jsonl').write_text('\n'.join(json.dumps(row) for row in report) + '\n')
    failures = [row for row in report if not row['ok']]
    print(f'cases={len(cases)} runs={len(report)} failures={len(failures)}')
    return 1 if failures else 0

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--cases', default='cases.jsonl')
    parser.add_argument('--primary-base', required=True)
    parser.add_argument('--primary-model', required=True)
    parser.add_argument('--shadow-base', required=True)
    parser.add_argument('--shadow-model', required=True)
    args = parser.parse_args()
    routes = [
        ('primary', args.primary_base, args.primary_model),
        ('shadow', args.shadow_base, args.shadow_model),
    ]
    raise SystemExit(run(args.cases, routes))

Run it with two environment keys and four route arguments:

export PRIMARY_API_KEY='...'
export SHADOW_API_KEY='...'
python shadow_contract.py --cases cases.jsonl --primary-base 'https://primary.internal/v1' --primary-model 'primary-model' --shadow-base 'https://shadow.example/v1' --shadow-model 'shadow-model'

Each line in cases.jsonl needs id, user, optional system, optional temperature, optional max_tokens, and optional timeout_s. Build 5-10 cases, not one demo prompt.

case id	purpose	max_tokens
sql_no_placeholder	confirm it follows a no-interpolation SQL instruction	160
json_only	request JSON only, then parse with jq	160
empty_reply	request one word; catches verbose mode and token waste	8
long_context	summarize one paragraph; detects silent context truncation	96
unicode_path	explain handling a file path containing spaces	128

What the report exposes

The artifact is the generated shadow_report.jsonl, not the console output. It makes four failure modes explicit.

• A 429 on shadow means the free tier has a rate limit or quota boundary that the demo did not reach.
• A 200 with empty choices means the provider returned success without a completion. Production code that assumes choices[0] will crash.
• Missing usage means you cannot audit token accounting, so a free token claim remains unmeasured.
• High first-case latency usually means cold start on the free server. Separate the first request from warm p95 before deciding whether the route is acceptable.

An example decision table:

observation	primary	shadow	decision
status 200, choices present, usage present	yes	yes	promote only after a repeat run
status 200, choices present, usage missing	yes	no	fail token accounting check
status 429	no	yes	keep shadow out of CI
first request 900 ms, warm p95 130 ms	170 ms	130 ms	acceptable for offline runs, not interactive use

These numbers are placeholders, not measurements from a specific endpoint. The point is to keep the decision threshold explicit and repeatable.

Limitations and who should not use this

The free route is useful for offline evaluation, not as a failover for real users. It may change models, quotas, server location, or availability without notice. A free server can be shared, can sleep, and usually has no SLA. Do not send personal data, secrets, or private code snippets to a free endpoint unless you have reviewed the data handling terms.

Do not use this workflow if you need contractual support, regulated data residency, strict latency guarantees, or a production fallback. The harness also measures the response envelope and a few output properties; it is not a correctness eval, and provider-reported token usage can be delayed or inaccurate.

Reusable checklist

1. Keep primary and shadow behind the same response shape.
2. Commit the static cases instead of reusing one chat transcript.
3. Record status, latency, prompt tokens, completion tokens, and error text for every case.
4. Separate first-call latency from warm latency.
5. Fail the merge on 429, empty choices, missing usage, or non-JSON bodies.
6. Set a hard timeout and a token ceiling before the run starts.
7. Store the JSONL report next to the run, not only in CI logs.
8. Do not put secrets or personal data in the cases file.

If you run this against the free route you are evaluating, keep the generated shadow_report.jsonl. The first cold-start number and the first quota error are the two data points that matter more than the marketing page.

Which part of your completion response is least stable in practice: the usage object, the choices array, or the HTTP status code? A failure there is usually the real merge blocker.