Score the Agent Before You Pick the Endpoint

You should score a coding agent on residency, traffic shape, eval fidelity, and blast radius before you pick any endpoint. Most teams skip that worksheet, drop a key into CI, and notice the mismatch only after a bot invents a file. I treat placement as an architecture choice, not a convenience default, because a cheap answer can still be an expensive incident. Why would a free hosted model be the right home for one loop and the wrong home for the next?

Placement is the decision, not the brand name

Agentic coding loops fail in boring ways when the runtime does not match the job. A pull request bot that sees private modules should not share a prompt cache with strangers, and an overnight refactor can tolerate queueing that an IDE chat cannot. I keep seeing people argue about model quality while the real leak is context leaving the building, or traces that cannot be replayed later. Are you actually choosing a model, or are you choosing who can read the repository?

I use four axes because they change operations, not marketing copy on a landing page. Data residency covers source, secrets adjacent to prompts, and whether traces are stored off-disk. Traffic shape covers interactive latency versus soak jobs that can wait without paging anyone. Eval fidelity covers whether you can replay a patch against a fixture instead of a vibe. Blast radius covers how hard a wrong edit is to unwind when git history is your only undo button.

This article is a proposed scorecard, not a production postmortem from a named company with invented metrics. I will give you a YAML profile, a small Python scorer, a unit test that locks the residency gate, and a mapping table you can argue with. Run the numbers on your own workload before you commit a bot to a free hosted box, a paid API, or a machine you operate yourself. If the total says stay local, believe it even when a free tier is sitting unused in another tab.

Axis notes I actually argue about

Residency (0–4, higher means safer to share). Score zero if prompts include customer data, private packages, or infrastructure diagrams that do not belong on a shared box. Score two if the corpus is internal but already mirrored to a vendor under a contract you have read. Score four only when the agent sees public docs and synthetic fixtures that you could paste into a gist. Would you paste that diff into a random Discord channel without flinching first?

Traffic shape (0–4, higher means delay is acceptable). Interactive IDE chat wants single-digit seconds, and CI comments can wait a minute without anyone filing a ticket. Overnight refactors can sit in a queue without paging a human who is already asleep. I score soak jobs high because a free shared server often bursts, then stalls, then recovers when the office goes quiet. Do you need a tight p95, or do you need the job to finish before morning standup?

Eval fidelity (0–4, higher means a shared endpoint is tolerable). If you cannot replay the tool trace, you cannot tell model drift from a busy neighbor on the same box. Fixture-based patch tests raise the score; "looks good in chat" keeps the number near zero on purpose. Cheap generation without a replay harness just manufactures technical debt faster than a human reviewer can catch it. What happens to that debt when the agent assumes a module exists and writes an import anyway?

Blast radius (0–4, higher means a bad answer is cheap). A docs typo is a four because revert is a five-minute conversation. A migration agent touching production schemas is a zero, even if the model is excellent on a blog benchmark you cannot reproduce. Pull request bots that can merge without review belong near zero even on a paid API with a serious-looking dashboard. Which failure can you undo with git revert before the standup coffee gets cold?

Numbered workflow I run before wiring tools

1. Write a workload profile instead of a slogan

Capture the agent as data, because slogans like "just use the free model" hide blast radius behind optimism. I keep a YAML file next to the bot so the next person can see why we did not put private source on a shared box. Is your profile honest about contains_private_source on the first draft, or did you answer for the workload you wish you had?

# profiles/pr-comment-bot.yaml
name: pr-comment-bot
prompts_include:
  - diff_hunks
  - failing_test_output
contains_customer_data: false
contains_private_source: true
interactive_latency_required: false
can_wait_minutes: true
has_replay_fixtures: true
can_merge_without_review: false
undo_with_git_revert: true
notes: "Comments only. No apply_patch on main."

2. Translate the profile into four integer scores

Do not average vibes across the team and call the result architecture. Map each boolean onto a 0–4 using the rubric above, and write the justification in a comment the next reviewer can attack. If two people disagree by more than one point on residency, the workload is not ready for a shared endpoint and you should stop. That disagreement is usually the most honest design document you will get all week.

3. Run the scorer on your laptop before you touch CI secrets

The script below is a proposed calculator, not a benchmark of any vendor, model, or hardware generation. It prints a placement band and the axis that dominated the veto, which is the number I actually bring to a review. If a required key is missing, it refuses to invent a recommendation, because a silent default is how bots wander onto the wrong box.

4. Map the total onto a placement band you can defend

I use three bands so the meeting cannot hide inside a foggy "it depends" and still ship a key. Low totals stay on a machine you control, including logs and prompt traces. Mid totals can use a paid API after you have actually read retention language. High totals are the only ones I even consider for a free hosted model and a free shared server.

5. Re-score after the first wrong patch lands

The first invented import is new evidence, not a funny anecdote for Slack. If the agent assumed a module that does not exist, drop eval fidelity by at least one point and run the file again. Placement is allowed to change after contact with a real repository, and it should change when the bot starts merging assumptions into comments. Are you scoring the model you wanted, or the blast radius you just observed?

A scorer you can run without a vendor SDK

Save this as score_agent_placement.py and point it at the YAML profile. It is ordinary Python three, and it keeps every axis visible so nobody can hide a residency veto inside an average.

#!/usr/bin/env python3
"""Proposed placement scorecard for coding-agent workloads."""
from __future__ import annotations

import argparse
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("Install pyyaml: pip install pyyaml\n")
    sys.exit(2)

RUBRIC = {
    "residency": "0=customer/PII or private source on a shared box; 4=public/synthetic only",
    "traffic": "0=interactive p95 required; 4=overnight soak is fine",
    "eval": "0=no replay fixtures; 4=patch replay + golden traces",
    "blast": "0=unreviewed prod mutation; 4=docs-only, easy revert",
}


def clamp(value: int) -> int:
    return max(0, min(4, int(value)))


def derive_scores(profile: dict) -> dict[str, int]:
    residency = 4
    if profile.get("contains_customer_data"):
        residency = 0
    elif profile.get("contains_private_source"):
        residency = 1
    traffic = 4 if profile.get("can_wait_minutes") else 1
    if profile.get("interactive_latency_required"):
        traffic = 0
    eval_s = 4 if profile.get("has_replay_fixtures") else 1
    blast = 4 if profile.get("undo_with_git_revert") else 1
    if profile.get("can_merge_without_review"):
        blast = 0
    return {
        "residency": clamp(residency),
        "traffic": clamp(traffic),
        "eval": clamp(eval_s),
        "blast": clamp(blast),
    }


def band(total: int) -> str:
    if total <= 8:
        return "control-plane-local: keep prompts on a machine you operate"
    if total <= 14:
        return "paid-api-with-logs: vendor OK if traces and retention are explicit"
    return "free-hosted-candidate: shared free models/server are eligible, not mandatory"


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--profile", required=True)
    args = parser.parse_args()
    raw = yaml.safe_load(Path(args.profile).read_text())
    required = [
        "contains_customer_data",
        "contains_private_source",
        "interactive_latency_required",
        "can_wait_minutes",
        "has_replay_fixtures",
        "can_merge_without_review",
        "undo_with_git_revert",
    ]
    missing = [k for k in required if k not in raw]
    if missing:
        sys.stderr.write(f"missing keys: {missing}\n")
        return 1
    scores = derive_scores(raw)
    total = sum(scores.values())
    print(f"workload: {raw.get('name', args.profile)}")
    for axis, value in scores.items():
        print(f"  {axis:10} {value}  # {RUBRIC[axis]}")
    print(f"total: {total}/16")
    print(f"band:  {band(total)}")
    weakest = min(scores, key=scores.get)
    print(f"gate:  {weakest} is the axis most likely to veto a free shared endpoint")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Commands I type when a new bot shows up in a pull request:

python3 -m pip install pyyaml pytest
python3 score_agent_placement.py --profile profiles/pr-comment-bot.yaml

Expected shape of the output, labeled as a fixture rather than a vendor benchmark you should cite:

workload: pr-comment-bot
  residency  1  # 0=customer/PII or private source on a shared box; 4=public/synthetic only
  traffic    4  # 0=interactive p95 required; 4=overnight soak is fine
  eval       4  # 0=no replay fixtures; 4=patch replay + golden traces
  blast      4  # 0=unreviewed prod mutation; 4=docs-only, easy revert
total: 13/16
band:  paid-api-with-logs: vendor OK if traces and retention are explicit
gate:  residency is the axis most likely to veto a free shared endpoint

Notice how a private source repo never becomes a free-hosted candidate on this rubric, even when traffic and eval look excellent on paper. That is the entire point of scoring residency separately from cost, latency, and the thrill of a free signup. If I flip contains_private_source to false and keep comments-only behavior, the total can cross into the free-hosted band without changing a single model name. Would you have caught that shift without writing the YAML down first?

Lock the gate with a test so a future refactor cannot "simplify" private source onto a shared box.

# tests/test_scorecard_gates.py
from score_agent_placement import band, derive_scores


def test_private_source_never_reaches_free_hosted_band():
    profile = {
        "contains_customer_data": False,
        "contains_private_source": True,
        "interactive_latency_required": False,
        "can_wait_minutes": True,
        "has_replay_fixtures": True,
        "can_merge_without_review": False,
        "undo_with_git_revert": True,
    }
    total = sum(derive_scores(profile).values())
    assert total <= 14
    assert "free-hosted" not in band(total)


def test_public_comment_bot_can_be_a_free_hosted_candidate():
    profile = {
        "contains_customer_data": False,
        "contains_private_source": False,
        "interactive_latency_required": False,
        "can_wait_minutes": True,
        "has_replay_fixtures": True,
        "can_merge_without_review": False,
        "undo_with_git_revert": True,
    }
    total = sum(derive_scores(profile).values())
    assert total >= 15
    assert band(total).startswith("free-hosted-candidate")

python3 -m pytest tests/test_scorecard_gates.py -q

How I read the three bands

Total	Band	What I do next
8 or below	Local / self-hosted	Run the agent runtime on a machine whose disk and logs I control.
9 to 14	Paid API with logs	Allowed only with retention, explicit trace export, and replay fixtures.
15 to 16	Free hosted candidate	Eligible for a shared free model and a free server, still behind the harness.

A candidate is not a mandate, and a low price does not move a veto axis by itself. I still refuse free shared inference when the gate axis is residency, even if the box is advertised as free and idle. Paid APIs fail the same way if you cannot export traces when a patch goes sideways at midnight. Self-hosting fails if nobody on the team will patch the box, and that operational cost belongs in the decision even when the electricity is already paid. Which of those failure modes does your team already know how to handle without a war room?

Where a free hosted option belongs in this matrix

When the scorecard lands in the high band, I want an endpoint I can try without standing up a GPU first or inventing a capacity plan. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option, which I treat as one eligible candidate after the numbers say a shared endpoint is acceptable. I do not put it in the path of a bot that can merge, and I do not assume any free tier is permanent, unlimited, or faster than a machine I already operate.

The useful comparison is operational, not sentimental, and it does not require a leaderboard of model names that will be stale next month. A free hosted model is a fit when prompts are non-sensitive, jobs can wait, fixtures exist, and a bad answer is cheap to revert in git. A paid API is a fit when you need a contract, an SLA conversation, and audit logs, not because the landing page looks serious in a screenshot. Self-hosting is a fit when residency is the veto, or when you already have people who enjoy operating models at inconvenient hours. If you cannot name the veto axis out loud, you are still shopping with vibes and a credit card nearby.

I wire the chosen endpoint behind the same client wrapper so the scorecard, not the SDK import, is what changes between experiments. That keeps retry storms and token buckets out of this article, because those are different design problems I have already poked at elsewhere. Here the only question is whether the agent is allowed to leave your disk with this particular workload shape.

Limitations, and who should not use this

This scorecard is ordinal, not scientific, and it will not satisfy a compliance questionnaire by itself no matter how tidy the YAML looks. It does not measure tokens, dollars, or answer quality, and it does not name models because those labels go stale faster than your profile file. Free hosted capacity can change without notice, so I never encode "always free" into a runbook or a dashboard alert. If you need guaranteed throughput, this worksheet should push you toward paid or local capacity, not toward hope and a status page you do not control.

Do not use this approach when prompts carry regulated data, when an agent can apply patches to production, or when you lack replay fixtures for the last bad edit. Do not use it as permission to skip a security review, because a four-number total is not a threat model. Do not use it if your agent is an unsupervised merge bot, because blast radius is already zero and the band will tell you to stay local. If two reviewers cannot agree on residency, stop and fix the data flow before you pick any vendor, free or otherwise.

I also would not use a four-number total for interactive IDE assistants that developers paste secrets into without noticing the transcript. That human channel needs a different control, usually shorter context and a local default that never leaves the laptop. The scorecard assumes you already know what the agent is allowed to read, which is a generous assumption on a busy Monday.

Commit the profile next to the bot, run the scorer as a lint, and fail the build when someone adds apply_patch without dropping the blast score. That is the whole practice, and it still works if you never try a hosted free tier at all. If your totals sit in the free-hosted band and you want one candidate that matches that band, run MonkeyCode's free models and free server against the same fixtures you already trust, then keep the YAML because the next agent will try to skip the worksheet.