A Two-Hour Fit Test for AI Coding Models on Your Own Codebase

Model recommendations usually arrive as screenshots: a clean demo, a bold claim, a comment section full of converted skeptics. The failure mode shows up later, in a boring pull request, when the model confidently renames an internal helper that forty files still import, or rewrites a Django view as if the project were Flask.

I stopped treating that as a model-quality problem and started treating it as a fit problem. The useful question is not “Is it smart?” but “Does it lower review risk in this repository, under our constraints, at our volume?”

That can be tested in about two hours with artifacts you already have: git history, review comments, CI logs, and a throwaway endpoint.

Start from evidence inside the repo

Do not invent toy prompts. Mine the last month of work for tasks that actually caused friction:

# Candidate friction points: files touched often, reverted commits, fixup chains
git log --since="45 days ago" --name-only --pretty=format: | sort | uniq -c | sort -nr | head -30
git log --since="45 days ago" --grep='revert\|fixup\|hotfix\|follow-up' --oneline

Turn what you find into eight “task cards,” each small enough to run in one shot. I use four lanes so the test is not just codegen:

1. Explain: “What invariant does billing/retry_policy.py protect, and where is it enforced?”
2. Patch: “Change pagination from offset to cursor without touching the public response shape.”
3. Test: “Write the regression test that would have caught PR #1187.”
4. Migrate: “Move this endpoint from the legacy validator to schema_v2, preserving error codes.”

Each card gets a pass condition before any model sees it. If you cannot write the pass condition, the task is not ready; it is a vibe.

A tiny runner, deliberately boring

Keep the harness small so the experiment is about the repo, not the framework. This sketch posts prompt-plus-context to an OpenAI-style endpoint and stores raw output for review. Treat it as a scaffold, not a finished product.

# fit.py -- scaffold, not production code
import json, os, pathlib, subprocess, time, urllib.request

ENDPOINT = os.environ["FIT_URL"]          # e.g. local proxy or approved gateway
KEY = os.environ.get("FIT_KEY", "")

def ask(prompt):
    body = json.dumps({
        "model": os.environ.get("FIT_MODEL", "unknown"),
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        **({"Authorization": f"Bearer {KEY}"} if KEY else {})
    })
    t = time.monotonic()
    with urllib.request.urlopen(req, timeout=120) as r:
        data = json.loads(r.read())
    return data["choices"][0]["message"]["content"], round(time.monotonic() - t, 2)

for card in sorted(pathlib.Path("cards").glob("*.md")):
    prompt = card.read_text()
    try:
        out, secs = ask(prompt)
        record = {"card": card.name, "seconds": secs, "output": out}
    except Exception as e:
        record = {"card": card.name, "error": repr(e)}
    with open("results.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

# Optional mechanical gate: apply suggested patch in a throwaway worktree.
# subprocess.run(["git", "worktree", "add", "/tmp/model-fit", "HEAD"], check=True)

The mechanical part matters most where code is produced: apply the suggestion in a disposable worktree, run the narrow test file, run the linter that actually gates CI, and record whether review would get easier or noisier. For explanations and design notes, use a three-line rubric: names real symbols, cites the right files, leaves an actionable next step. Anything that invents APIs gets zero, not partial credit.

Where free access fits without becoming a dependency

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's current pitch includes free model access and a free server option, which can make it a convenient target for a first run of this harness: no local GPU, and no procurement step before you learn whether your cards are even well designed. I am not claiming specific models, quotas, latency, retention, or how long either option remains available; check the live docs and your own approval path before pointing real code at it.

The free tier is best used as a measurement lab. It can answer “Is my task suite sensitive enough?” and “Which lanes are trivially solved?” before you spend money or political capital. It should not quietly become infrastructure. If a passing score starts to matter, write the exit criteria next to the scorecard.

Decide with gates, not averages

A single mean score hides the failures that dominate review time. I use a small decision table and require every gate to be answered before the quality number is discussed.

Gate / signal	Question to answer	Evidence
Data path	Which snippets leave the machine, and is that allowed?	Proxy logs, endpoint docs, security note
Repeatability	Same card, three cold runs: stable or slot machine?	results.jsonl across runs
Blast radius	When wrong, does it fail loudly or merge quietly?	Patch applied in worktree, CI subset
Review delta	Fewer comments, same comments, or new categories of nit?	Sample PR review after use
Cost at volume	What happens at 10x and 100x current usage?	Your usage estimate, not vendor math
Exit plan	Can we swap endpoints by changing env vars only?	One-flag retest script

Notice what is absent: a universal ranking. A suite built from your history can rank options for you; it cannot support claims about models in general. Publishing it as a benchmark would overclaim and probably age badly.

Limits, and who should skip this

• One-shot prompting only. Agent loops, tool use, and multi-file edits need a different rig.
• Eight cards can expose mismatch; they cannot prove safety.
• Free access is a weather report. Limits, terms, and availability move.
• If code cannot leave your environment, do not “just try it” on an outside server.
• If your organization already has an approved toolchain, test inside that list. Your constraint is governance, not curiosity.
• If the hard part is flaky requirements rather than model output, better prompts will not rescue the process.

The durable win is not finding a bargain model. It is owning a repeatable way to ask, “Did this change reduce the risk I personally have to review?” Once that harness exists, a new endpoint is an environment variable and an afternoon, not a migration.

If you build a card set from your own git history, I would be curious which lane surprised you most: explanation, patching, tests, or migration.