Case Study: A Golden-Set Gate That Lets a Free Model Tier Serve One Boring CI Job

Short answer up front: you can move one narrow, non-customer-facing job onto a free model tier, provided you freeze the output contract first and gate every swap with a golden set. This case study walks through a CI log-triage CLI, the small harness that decides which model may serve it, and the parts I intentionally kept manual.

Background: one boring job with an unusually strict contract

triage reads a failed CI job's raw log and prints exactly one bucket, one short reason, and one evidence line number. The buckets are deliberately boring: flaky_test, missing_env, dep_conflict, oom, timeout, permission, and unknown. A discussion bot posts that result into the pull request thread, so a malformed object is more damaging than a slightly wrong label. The job had been running against a paid API at low but non-zero volume, which is precisely the profile worth testing on a free tier.

MonkeyCode is an open-source project offering free model access and a free server option, both of which the operator describes as current parts of its offering. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am deliberately not quoting a token allowance or any machine detail here, because those change often and a stale number in a tutorial is worse than no number at all. Check the project's live terms before you size anything around it.

Goal, and the things I refused to change

The goal was narrow: keep triage's stdout contract byte-identical while the model behind it became negotiable.

• One JSON object on stdout, exactly three keys, exit code 0 on success.
• Every unknown bucket still routes to a human, because that is where silent wrongness hides.
• A model swap must be a reviewable diff rather than a runtime experiment.
• The whole check has to finish in under a minute so it can run on every push.

The non-goals matter just as much, because scope creep is how a free-tier experiment turns into an incident.

• No streaming, no chat surface, and no multi-turn agent loop around the classifier.
• No claim of general parity with the paid model, only parity on this one frozen task.
• No automatic promotion of a new model; a human merges that change on purpose.

Decision table: which jobs deserve a free tier

Use this as a filter before you write any adapter code at all.

Job	Customer-visible	Output strictness	Move it?
CI log triage	No, internal bot only	One enum plus two fields	Yes
PR title cleanup	No	Free-form text	Yes, if nobody blocks on it
Release notes draft	Yes, public changelog	Tone plus accuracy	No, keep a human reviewer
Support reply generation	Yes, sent to customers	Legal and brand risk	No

The pattern is simple: strict, machine-checkable output plus an internal audience is where a free tier earns its place.

The artifact: a golden set and a small gate

The harness is the durable part of this project, not the prompt. Set up the layout first, then keep every number in version control next to the code.

triage/
├── adapter.py       # the only file that knows about a provider
├── golden.py        # scoring plus the swap gate
├── baseline.json    # the accepted score of the current model
└── golden/
    ├── 01-timeout.log
    ├── 01-timeout.expected.json
    └── ...

Step 1: freeze the contract in code

Write the validator before you talk to any model, because it becomes the definition of done for the whole task.

#!/usr/bin/env python3
# golden.py — score a model against a frozen contract, then gate the swap.

BUCKETS = ['dep_conflict', 'flaky_test', 'missing_env', 'oom',
           'permission', 'timeout', 'unknown']

def validate(obj):
    # Return None when the object satisfies the contract, else a short reason.
    if not isinstance(obj, dict):
        return 'not_object'
    if set(obj) != {'bucket', 'reason', 'evidence_line'}:
        return 'keys'
    if obj['bucket'] not in BUCKETS:
        return 'bucket_enum'
    reason = obj['reason']
    if not isinstance(reason, str) or not reason.strip():
        return 'reason_empty'
    if len(reason) > 160:
        return 'reason_long'
    line = obj['evidence_line']
    if not isinstance(line, int) or isinstance(line, bool) or line < 1:
        return 'evidence_line'
    return None

Step 2: build a small golden set, including impossible cases

Thirty anonymized logs were enough for this task, but five of them were written to be genuinely ambiguous on purpose.

• Twenty-five logs with one obvious cause and a stable expected bucket.
• Five logs where two causes are equally plausible, expected bucket unknown.
• The expected answer stored as JSON next to each log and reviewed like a fixture.
• Hostnames, tokens, and repository names anonymized before anything leaves the machine.

Those five ambiguous rows are the most valuable entries in the set, because they catch a model that guesses confidently.

Step 3: keep exactly one provider-aware file

Everything that knows about a network endpoint lives in adapter.py, and the shape below is illustrative, so match your provider's actual API.

# adapter.py — keep this file small and boring.
import os, json, urllib.request

def complete(prompt: str, timeout: int = 60) -> str:
    # Shape below is illustrative; match your provider's real request format.
    req = urllib.request.Request(
        os.environ['FREE_MODEL_URL'],
        data=json.dumps({'prompt': prompt}).encode(),
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + os.environ['FREE_MODEL_KEY']},
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read())['text']

Step 4: score schema first, accuracy second

Schema failures are cheap to measure and expensive to ignore, so they belong first in the report.

import json

PROMPT = '''You classify CI failure logs.
Return one JSON object only, no prose and no markdown fence.
Keys: bucket, reason (160 chars max), evidence_line (integer, 1 or more).
If two explanations are equally likely, use the bucket unknown.

LOG:
{log}'''

def call_model(text):
    raw = complete(PROMPT.format(log=text)).strip()
    if raw.startswith('```

'):                       # models love fences
        raw = raw.split('\n', 1)[1].rsplit('

```', 1)[0]
    return json.loads(raw)

def score(cases):
    failures, bucket_ok, abstain_ok, abstain_total = [], 0, 0, 0
    for path, expected in cases:
        try:
            obj = call_model(path.read_text())
        except Exception as exc:                    # network, JSON, timeout
            failures.append((path.name, 'call:' + type(exc).__name__))
            continue
        bad = validate(obj)
        if bad:
            failures.append((path.name, bad))
            continue
        bucket_ok += obj['bucket'] == expected['bucket']
        if expected['bucket'] == 'unknown':
            abstain_total += 1
            abstain_ok += obj['bucket'] == 'unknown'
    return {
        'cases': len(cases),
        'schema_pass': 1 - len(failures) / len(cases),
        'bucket_acc': bucket_ok / len(cases),
        'abstain_recall': abstain_ok / abstain_total if abstain_total else 1.0,
        'failures': failures,
    }

Step 5: make the gate exit non-zero

A gate that only prints numbers is a dashboard, and dashboards get ignored during a busy week.

GATE = {'schema_pass': 1.00, 'bucket_drop': 0.05, 'abstain_recall': 0.80}

def verdict(result, baseline):
    reasons = []
    if result['schema_pass'] < GATE['schema_pass']:
        reasons.append('schema_pass below 1.00')
    if result['bucket_acc'] < baseline['bucket_acc'] - GATE['bucket_drop']:
        reasons.append('bucket_acc dropped more than 0.05')
    if result['abstain_recall'] < GATE['abstain_recall']:
        reasons.append('abstain_recall below 0.80')
    return reasons

Wire verdict into main, write the raw scores into results/<date>.json, and exit with code 1 whenever that reason list is non-empty.

Running the gate on the free server instead of a laptop

A check that only runs on your laptop is a check that stops running the week you travel. The operator's free server option fits this stage well, because the harness needs no GPU, no private data, and no long-lived state beyond a results directory.

# nightly, on whatever host you already trust
python golden.py --baseline baseline.json --out results/$(date -u +%F).json
echo exit=$?   # 0 means the current model still passes

Keep the golden logs anonymized, keep the results inside the repository, and never let the nightly job swap the model by itself.

Results: what the harness actually decides

The exact figures below are an illustrative dry run, so reproduce them against your own baseline before you trust any of it.

Signal	Illustrative dry run	Gate rule
schema_pass	1.00	must equal 1.00
bucket_acc	baseline 0.83, candidate 0.80	no worse than baseline minus 0.05
abstain_recall	0.80, four of five ambiguous logs	at least 0.80
wall clock	roughly forty seconds for thirty logs	under one minute

The useful outcome is not the score itself, but the failure list. Most rejected candidates failed on keys or reason_long rather than on the bucket label, which means prompt hardening, not a better model, was the real fix.

Lessons learned

1. Freeze the contract before you shop around, because a moving target cannot be evaluated fairly.
2. Score schema conformance first, since a malformed object breaks automation more quietly than a wrong label.
3. Keep at least five impossible cases, because confident guessing is the failure mode that hurts most.
4. Confine provider knowledge to one small file so a swap never touches business logic.
5. Make the gate exit non-zero, because an advisory check is a suggestion and suggestions lose to deadlines.

Who should not use this approach

Skip it if the output is customer-visible prose, if a wrong answer carries legal or financial blast radius, or if nobody will maintain the labeled examples over time. Skip it as well for latency-critical paths, where a free tier may not offer the tail behavior you need. The method fits internal, boring, machine-checkable jobs and very little else.

A closing note on where this fits

If you already keep a golden set for a paid model, pointing that same harness at a free tier costs one afternoon and buys you a real answer about your own contract. Run your own numbers on your own logs, then let the gate make the call instead of your optimism.