The Swap Harness: What a Senior Asked Before the Team Trusted a Free-Tier Coding Assistant

The decision that survived the pairing session was narrow and slightly boring: a free-tier coding assistant may draft code and test scaffolding for this team, but nothing it produces reaches a merge request until it clears a swap harness first. The harness is about eighty lines of Python, one manifest writer, and a decision table with abandon criteria written down before any prompt was sent. It exists because the senior on the call refused to argue about model quality in prose and asked for something cheaper than an opinion.

The assistant under evaluation was MonkeyCode, an open-source coding assistant project whose operator states that it offers free model access along with a free server option for running it. The same outreach materials describe a free allowance on the order of ten million tokens, which is a figure that moves with plan changes and therefore has to be re-checked against current documentation on the day it actually matters.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

No model names, latency figures, throughput numbers, or acceptance rates appear below, because the entire point of the workflow is to survive alias changes and because inventing those numbers would be worse than omitting them. Everything here is a method plus an adapter seam the reader fills in.

The three questions that set the agenda

The senior opened with three reported questions rather than a verdict, and each one collapsed a category of vague complaint into something testable. The first was which part of a generated artifact the team actually verifies, since reviewing a diff by eye is not verification. The second was what specific observation would make the team stop using the free tier, asked before any enthusiasm had accumulated. The third was who signs off on a run that nobody can reproduce next month.

Those three questions produced three artifacts, in order: a predicate list per case, an abandon-criteria column, and a run manifest. The pairing session then spent the remaining hour deleting anything that did not feed one of them.

Three dead ends, hit in the order they occurred

The first dead end was the eyeball benchmark, where the same prompt went to two backends and the pair compared the outputs side by side. It felt informative and produced nothing durable, because a sample size of one with no recorded prompt hash cannot be re-examined or disputed later.

The second dead end was pinning a model alias in the README as if documentation were a lockfile. Aliases drift, endpoints get repointed, and a reader six weeks later inherits a claim with no receipt attached to it.

The third dead end was letting the test file call a vendor client directly, which guarantees that every future swap becomes a refactor instead of a configuration change. The pair kept the adapter seam and threw away the rest of that afternoon's work.

Step 1: Write the run manifest before writing any prompt

The manifest is the artifact that makes a later conversation possible, and it costs about thirty lines. It records what ran, against which host and alias, on which commit, at which time, so that a green result cannot be silently attributed to the wrong backend.

# manifest.py — requires Python 3.11+ for datetime.UTC
import datetime
import hashlib
import json
import os
import platform
import subprocess


def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def build_manifest(prompt: str, provider: dict, suite: str) -> dict:
    return {
        "suite": suite,
        "prompt_sha": sha(prompt),
        "prompt_chars": len(prompt),
        "provider_label": provider["label"],
        "endpoint_host": provider["host"],
        "model_alias": provider["alias"],
        "python": platform.python_version(),
        "platform": platform.platform(),
        "git_sha": subprocess.check_output(
            ["git", "rev-parse", "--short", "HEAD"], text=True
        ).strip(),
        "recorded_at_utc": datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"),
    }


def write_manifest(manifest: dict, path: str) -> None:
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(manifest, fh, indent=2, sort_keys=True)

Step 2: Run the same cases against two backends

The suite is a parameterized pytest file where providers come from environment variables and cases come from a JSON file. The adapter and checker fixtures are deliberately not shown, because they are the only vendor-specific part and the reader owns them.

# tests/test_swap_conformance.py
import json
import os

import pytest

from manifest import build_manifest, write_manifest

PROVIDERS = [
    {"label": "free-tier", "host": os.environ.get("FREE_HOST", "unset"),
     "alias": os.environ.get("FREE_ALIAS", "unset")},
    {"label": "baseline", "host": os.environ.get("BASE_HOST", "unset"),
     "alias": os.environ.get("BASE_ALIAS", "unset")},
]

CASES = json.load(open("cases/drafting.json", encoding="utf-8"))


@pytest.mark.parametrize("provider", PROVIDERS, ids=[p["label"] for p in PROVIDERS])
@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_swap_conformance(provider, case, adapter, checker):
    if provider["host"] == "unset":
        pytest.skip(f"{provider['label']} endpoint is not configured")
    manifest = build_manifest(case["prompt"], provider, suite="drafting")
    write_manifest(manifest, f"artifacts/{case['id']}-{provider['label']}.json")
    raw = adapter.call(provider, case["prompt"])
    normalized = adapter.normalize(raw)
    assert checker(normalized, case["expect"]), (
        f"{case['id']} failed on {provider['label']}; prompt_sha={manifest['prompt_sha']}"
    )

Cases carry structural predicates instead of prose expectations, which keeps them stable across phrasing changes. A representative case file looks like this:

[
  {
    "id": "retry-helper",
    "prompt": "Write a Python function retry(fn, attempts=3) that retries on exception and raises the last error.",
    "expect": {
      "must_include": ["def retry", "raise"],
      "must_not_include": ["except:", "time.sleep(0)"],
      "max_lines": 40
    }
  }
]

The run itself is two exports and one command, and the artifacts directory becomes the evidence trail. A run with an unset host skips loudly rather than passing quietly, which the senior insisted on.

export FREE_HOST=<free-endpoint-host> FREE_ALIAS=<alias-from-current-docs>
export BASE_HOST=<comparison-host>   BASE_ALIAS=<alias-from-current-docs>

python -m pytest tests/test_swap_conformance.py -q --maxfail=1
ls artifacts/ | head

Step 3: Normalize before comparing, never after

Normalization lives inside the adapter, so no comparison code ever sees raw model output. Three rules cover most of the noise: strip fence language tags and surrounding whitespace, case-fold identifiers used in substring predicates, and sort import lines before line counting. Nothing here judges quality, and that limitation is intentional rather than an oversight.

Patching a checker after seeing a failure is the fastest way to turn a conformance suite into a rubber stamp. The pair's rule was that a predicate change requires a separate commit with its own justification, which makes the temptation visible in review.

The decision table the pair kept

Observation	Action
Free tier passes all predicates and the failure set matches the baseline	Keep the free tier for drafting and scaffolding
Free tier fails a predicate that the baseline passes	Open an issue with the manifest path and the prompt hash
Both backends fail the same predicate	The predicate is wrong; fix the case file first
A run cannot be reproduced from its manifest	Freeze adoption; the comparison is not evidence
The free allowance or server option changes	Re-run the suite before trusting prior results
Any case requires human reading to judge	Move it out of the suite and into review

Rows four and six carried the session, because they define exits rather than endorsements. A team that cannot reproduce a run has no basis for a decision, and a case that needs human taste does not belong in an automated gate.

The decision the pair actually kept

The kept decision was scoped to drafting and test scaffolding, with a written boundary that production logic stays under human review regardless of how green the suite looks. The senior's framing was that a passing harness proves consistency across endpoints, not correctness of the code under discussion, and the team adopted that sentence verbatim in the ADR.

The second kept item was the abandon criterion, which named two triggers: repetitive predicate failures on the free tier and any change in availability terms that cannot be re-verified from primary documentation. Writing the exit before the enthusiasm is what made the adoption reversible.

Limitations and who should not use this approach

The harness measures conformance to hand-written predicates and nothing else, so it says little about readability, security posture, or fitness for a domain the predicates do not cover. Manifests also have a shelf life, and a run recorded against an alias that no longer exists is archaeology rather than evidence.

Teams handling data that cannot leave a defined boundary should not route it through any third-party endpoint, free or otherwise. Teams that need bit-for-bit deterministic reproduction should not adopt this at all, since sampling behavior alone breaks that requirement. Anyone who would read a green suite as permission to merge without review has misunderstood the artifact.

A single afternoon with a throwaway copy of this harness is the cheapest way to test whether MonkeyCode's free model access and free server option fit an existing review process, and the manifest files that come out of it are more persuasive to a skeptical senior than any amount of prose.