The Demotion Runbook: Taking a Workload Off a Free Model Without a Bad Deploy

The night the free lane stopped being free

A nightly summarizer sat on a free model lane for five months. Its eval suite passed every Friday. Then an upstream schema change doubled the average input length. Retry volume tripled before Monday. A reviewer burned two hours re-reading outputs nobody trusted.

The team had an onboarding checklist. It had no removal plan. The scenario above is a composite illustration, not a customer report.

A passing eval is not a residency permit

• Evals measure answer quality on a frozen sample.
• They do not measure retry amplification, reviewer minutes, or blast radius.
• They do not expire, so a green suite quietly becomes a permission slip.

The pattern shows up in current discussion too. One recent DEV post reports getting an eval suite to zero mistakes and still declining to make that model the default. That instinct is correct. A clean score backs a model choice. It does not license a permanent deployment.

Three tripwires that force a demotion decision

Tripwire 1: retry amplification

Count retries and divide by first attempts. A lane above 1.5 is telling you the input distribution moved. Paying for a stronger model is often cheaper than paying for retries.

Tripwire 2: reviewer minutes per 1,000 attempts

Track how long a human spends correcting or re-reading output. This number hides well. It rarely appears in a cost dashboard, and it is usually the first tripwire to fire.

Tripwire 3: blast radius

Count distinct downstream writers the lane touches. Two writers means a demotion is a config change. Five writers means a demotion is an incident.

The artifact: a lane ledger and tripwire gate

Save this as lane_ledger.py. It reads a JSONL event log, computes the three tripwires per lane, and returns exit code 1 when any lane must be demoted.

#!/usr/bin/env python3
"""lane_ledger.py — tripwire check for workloads parked on a free model lane."""
import argparse
import json
import sys
from collections import defaultdict
from dataclasses import dataclass


@dataclass
class Tripwires:
    retry_amp: float = 1.5           # retries / first attempts
    review_min_per_1k: float = 20.0  # reviewer minutes per 1k attempts
    blast_radius: int = 3            # distinct downstream writers


def load(path):
    with open(path, encoding="utf-8") as fh:
        for n, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                yield json.loads(line)
            except json.JSONDecodeError as exc:
                raise SystemExit(f"bad JSON on line {n}: {exc}")


def summarize(events):
    lanes = defaultdict(lambda: {"attempts": 0, "retries": 0, "review_min": 0.0,
                                 "writers": set(), "rejects": 0, "tokens": 0})
    for ev in events:
        row = lanes[ev["lane"]]
        kind = ev["kind"]
        if kind == "attempt":
            row["attempts"] += 1
        elif kind == "retry":
            row["retries"] += 1
        elif kind == "review":
            row["review_min"] += float(ev["minutes"])
        elif kind == "write":
            row["writers"].add(ev["target"])
        elif kind == "reject":
            row["rejects"] += 1
        else:
            raise SystemExit(f"unknown kind: {kind}")
        row["tokens"] += int(ev.get("tokens", 0))
    return lanes


def evaluate(row, t):
    attempts = row["attempts"] or 1
    amp = row["retries"] / attempts
    review = row["review_min"] * 1000 / attempts
    blast = len(row["writers"])
    hits = []
    if amp > t.retry_amp:
        hits.append(f"retry amplification {amp:.2f} exceeds {t.retry_amp}")
    if review > t.review_min_per_1k:
        hits.append(f"review load {review:.1f} min/1k exceeds {t.review_min_per_1k}")
    if blast >= t.blast_radius:
        hits.append(f"blast radius {blast} writers reaches {t.blast_radius}")
    return ("DEMOTE" if hits else "STAY"), hits, amp, review, blast


def main(argv):
    ap = argparse.ArgumentParser(description="Free-lane tripwire check")
    ap.add_argument("ledger", help="JSONL file of lane events")
    ap.add_argument("--retry-amp", type=float, default=1.5)
    ap.add_argument("--review-min-per-1k", type=float, default=20.0)
    ap.add_argument("--blast-radius", type=int, default=3)
    args = ap.parse_args(argv)
    t = Tripwires(args.retry_amp, args.review_min_per_1k, args.blast_radius)
    lanes = summarize(load(args.ledger))
    print(f"{'lane':<20}{'amp':>6}{'rev/1k':>9}{'blast':>7}  verdict")
    demoted = False
    for name, row in sorted(lanes.items()):
        verdict, hits, amp, review, blast = evaluate(row, t)
        print(f"{name:<20}{amp:>6.2f}{review:>9.1f}{blast:>7}  {verdict}")
        for hit in hits:
            print(f"    - {hit}")
        demoted = demoted or verdict == "DEMOTE"
    return 1 if demoted else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Record events from your own pipeline. One line per event:

{"lane":"free-summarizer","kind":"attempt","tokens":820}
{"lane":"free-summarizer","kind":"retry","tokens":810}
{"lane":"free-summarizer","kind":"review","minutes":4.5}
{"lane":"free-summarizer","kind":"write","target":"s3://reports/daily"}
{"lane":"free-summarizer","kind":"write","target":"webhook:slack"}
{"lane":"free-summarizer","kind":"reject"}

Run it against a week of traffic:

python lane_ledger.py ledger.jsonl --retry-amp 1.3 --review-min-per-1k 15

Expected output shape:

lane                    amp   rev/1k  blast  verdict
free-summarizer       1.00   4500.0      2  DEMOTE
    - review load 4500.0 min/1k exceeds 15.0

The verdict is mechanical. The thresholds are not. Set them from your own service level, and commit the values so they are reviewable.

Reading the verdict

Verdict	Signal	Next move
STAY	All three tripwires clear	Keep the lane, re-run weekly
DEMOTE, review only	Quality holds, human cost does not	Add a deterministic pre-filter
DEMOTE, retry amp	Lane is unstable on real input	Move to a paid tier or a self-hosted model
DEMOTE, blast radius	Lane touches too many writers	Split it; keep read-only paths only
Two or more hits	The lane is load-bearing	Freeze feature work and schedule a cutover

Red flags: when not to use a free lane at all

1. The output gates a release, an eval report, or a billing event.
2. The workload signs, writes to, or acknowledges a signed webhook.
3. The workload holds or renews a lease, lock, or idempotency key.
4. A legal hold, retention rule, or audit requirement applies.
5. p99 latency is a product promise rather than a nice-to-have.
6. A retry storm would hit a shared dependency you do not control.
7. The cost of one wrong answer exceeds the cost of a paid call.

If two or more flags apply, the free lane is the wrong tier. No amount of prompt work fixes an architecture mismatch.

Better alternatives by failure mode

Failure mode	Cheaper fix than a stronger model
Invalid JSON output	Schema validation plus one bounded repair pass
Long inputs	Chunking with a deterministic reducer
Retry storms	Token-bucket queue and a dead-letter lane
Reviewer fatigue	Confidence threshold plus sampled human review
Slow tail latency	Batch the work and move it off the request path

Three rows in that table remove the model entirely. Free model access is a cost lever, not a design decision.

Write the exit criteria before you onboard

Four numbers belong in the design doc on day one.

1. Retry amplification ceiling, for example 1.5.
2. Reviewer minutes per 1,000 attempts, for example 20.
3. Blast radius cap, for example two writers.
4. A demotion deadline: two consecutive red weeks triggers the cutover.

Anyone can add a lane. Someone has to own removing it.

Where MonkeyCode fits, and where it does not

MonkeyCode is an open-source project that provides model access and a server option for developers. Per details supplied by the project's operator, the free tier includes 10 million free tokens and a free server. Free-tier terms change, so confirm current limits and duration in the project's own repository before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Under this runbook, MonkeyCode's free tier fits rehearsal rather than residency. Use it to run the ledger on throwaway lanes, prototype read-only paths, and practice the cutover before real traffic arrives. Do not use it for the red-flag list above. A lane that signs artifacts, renews leases, or gates a release needs a tier with an explicit contract.

The token budget is generous for prototyping. It is not a substitute for the four exit criteria. Cheap inference still costs reviewer minutes when the output is wrong.

Limitations

• The composite scenario above is illustrative and not a measured benchmark.
• The tripwire thresholds are examples, not industry standards.
• No latency, uptime, or retention guarantee is claimed for any tier.
• The ledger script covers three signals only. Add more only if you can act on each one.

One last thing

Run the ledger against your own traffic first. If the numbers clear your thresholds, MonkeyCode's free tier is a reasonable place to rehearse the next lane.