A Three-Way Residency Trial: Local-Only, Server-Only, and Split Fallback Agent Runs

Most residency arguments are settled with a single-arm benchmark, and that is why they keep getting re-litigated. A developer runs a task suite on a local model, then runs the same suite against a hosted free model, and compares two numbers that were never measuring the same thing. The local arm fails on truncation, the hosted arm fails on connect timeouts during a commute, and the comparison collapses into anecdote.

The scene that motivates a third arm

An illustrative composite: a small team keeps an agent on a laptop because the repository contains customer fixtures. Someone proposes moving the batch to a free server seat to escape fan noise and short tool loops. The move fixes throughput and breaks a review rule, so the work oscillates between two arms for weeks. Neither arm was ever tested with a fallback path, which is the configuration the team would actually ship.

MonkeyCode, an open-source coding agent project, is relevant here only because it exposes both a free model path and a free server option as operator-stated availability; the trial below measures whichever backends the reader supplies. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Confirm current free-tier terms on the project's own pages, since quotas and availability change without notice.

Why two arms produce unstable conclusions

A two-arm comparison answers "which was faster," while a shipping decision needs "which fails, and how." Failure class matters more than median latency because transport errors are retryable, schema errors are cheap to validate, and wrong answers cost human review minutes that never appear in a latency chart. The third arm, split with fallback, exposes which of those classes the fallback actually absorbs.

Step 1: Fix a small task suite with gradeable outputs

Twelve to twenty tasks is enough to start, provided every task has a validator that returns pass or fail without human judgement. Include at least four task shapes: a pure edit, a multi-file refactor, a tool-loop task such as running tests, and a task requiring a file the process must read from disk. The suite should stay frozen for the duration of the trial, because a shifting suite converts the measurement into a moving target.

Step 2: Instrument the transport, not the model

The harness must not care whether the backend is a local runtime or a remote seat. Wrap both behind one callable that returns an artifact plus a status, then record every attempt as one JSONL line. Recording at the attempt level, rather than the task level, is what later makes retries visible.

# trial.py — runnable against any adapter implementing run(artifact_spec) -> (text, status)
import argparse, json, pathlib, time, uuid
from dataclasses import dataclass, asdict

FAILURE_CLASSES = ("none", "transport", "schema", "tool_loop", "truncation", "wrong_answer")

@dataclass
class Attempt:
    run_id: str
    arm: str          # local | server | split
    task_id: str
    attempt: int
    backend: str      # resolved backend for this attempt
    wall_ms: int
    status: str       # ok | error | timeout
    tool_calls: int
    egress_bytes: int # payload size handed to a remote backend, 0 for local
    failure_class: str

def classify(status, validator_error, tool_calls, egress_bytes):
    if status == "timeout":
        return "transport"
    if status == "error":
        return "tool_loop" if tool_calls else "transport"
    if validator_error is None:
        return "none"
    if validator_error.startswith("truncated"):
        return "truncation"
    if validator_error.startswith("schema"):
        return "schema"
    return "wrong_answer"

def run_arm(arm, tasks, adapters, out_path):
    run_id = uuid.uuid4().hex[:12]
    with open(out_path, "a", encoding="utf-8") as sink:
        for task in tasks:
            order = adapters.order_for(arm, task)   # local, server, or both
            for i, backend in enumerate(order):
                started = time.monotonic()
                text, status, tool_calls, egress = adapters.call(backend, task)
                wall_ms = int((time.monotonic() - started) * 1000)
                err = task.validate(text)
                rec = Attempt(run_id, arm, task.id, i, backend, wall_ms,
                              status, tool_calls, egress,
                              classify(status, err, tool_calls, egress))
                sink.write(json.dumps(asdict(rec)) + "\n")
                if rec.failure_class == "none" or arm != "split":
                    break

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--arm", choices=("local", "server", "split"), required=True)
    p.add_argument("--suite", default="tasks/")
    p.add_argument("--out", default="runs.jsonl")
    a = p.parse_args()
    run_arm(a.arm, load_suite(a.suite), load_adapters(), a.out)

The ordering helper is the only arm-specific logic, and it stays deliberately boring: local-first for the local arm, server-first for the server arm, and local-then-server for split. Because the fallback in split is triggered by any non-pass validator result, the harness over-triggers on purpose; a production router would retry transport errors only.

Step 3: Run the three arms on the same machine

python trial.py --arm local  --out runs/local.jsonl
python trial.py --arm server --out runs/server.jsonl
python trial.py --arm split  --out runs/split.jsonl
python report.py runs/*.jsonl --markdown

Run all three from the same host and the same network conditions, and note the time of day next to each run. A server arm measured on office wifi and a local arm measured on tethered mobile data will produce a difference that reflects the network, not the residency.

Step 4: Read the failure distribution before the latency

The report should print, per arm, the share of each failure class and the median wall time for attempts that passed. The useful comparison is not local versus server; it is split versus the better of the two single arms. If split does not reduce the dominant failure class, the fallback is decorating the pipeline rather than protecting it.

Step 5: Convert the output into a routing table

Signal in the trial output	Default residency	Reason
Dominant class is transport	local	Every remote attempt pays a network it does not need
Dominant class is schema, local validator cheap	local	Retry loops stay off the wire and off the quota
Dominant class is tool_loop on local	server	Longer tool budgets usually live where the host does not sleep
egress_bytes above the review threshold	local or redacted	Residency follows the payload, not the benchmark
Offline requirement in the task spec	local	No fallback can recover an absent link
Long unattended batch, no secrets	server	Host sleep and lid close stop mattering

The table is a starting hypothesis, not a verdict, and it should be regenerated whenever the backend or the suite changes.

Limitations and who should skip this

A twelve-task suite supports a routing decision, not a statistical claim, and a single week of runs will not survive a model update. Teams whose artifacts cannot be graded automatically should not adopt the harness, because manual grading reintroduces the bias the trial was meant to remove. Regulated payloads need an egress review before any server arm is measured at all, and anyone expecting the free tier to remain static should treat the numbers as expiring on the day they are printed.

Closing note

Isolation and measurement beat enthusiasm in both directions: the local machine is not automatically safer, and the free server is not automatically faster. The project's repository documents its current free model access and free server option, and both are worth confirming against live terms before either is wired into a pipeline.