Where Should the Task Run? A Placement Glossary, a Four-Branch Tree, and a Worked Example at Every Leaf

The pattern is always the same shape. An agent loop works fine on one machine, then someone moves it to a shared runner, and three days later the nightly job is forty minutes slower and nobody can say which model call, which shell step, or which host did it. A pattern, not a specific incident.

Most agent debugging starts at the model. That is usually the wrong layer. Before you argue about prompts or retries, decide where the work executes, because placement determines cost, blast radius, and what you can even measure. This post gives you a small vocabulary, a four-branch decision tree, and a concrete probe you can run at each leaf.

A short placement glossary

Six terms carry most of the weight. Define them before you draw anything.

1. Route — the triple of (model endpoint, compute host, credential scope). A route is not a model. It is everything a step touches when it runs.
2. Placement — which route a given step uses. Placement is per-step, not per-project.
3. Context residency — where prompts, files, and tool output sit at rest. If residency changes, your review process changes with it.
4. Egress class — the categories of outbound traffic a step needs: none, package registry, git remote, or arbitrary external API. Broad classes widen blast radius.
5. Cold start — wall-clock time from leasing a host to the first successful model call. Cold start is where "it worked locally" usually dies.
6. Route ledger — the per-run record of route, budget spent, egress used, and outcome. Without a ledger, placement decisions are folklore.

Two candidate routes worth cataloguing

An open-source project in this space, MonkeyCode, is relevant here for one narrow reason: it publishes a route that removes two common blockers at once. The operator states that the project offers free model access and a free server option, so a developer can test placement decisions without first provisioning a host or funding an API key. Those are operator-supplied availability claims, not benchmarks I ran, and free allocations change — verify current terms in the project before you depend on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat it as one row in your own route catalog, not as the default answer. The tree below is what decides.

The four-branch tree

Ask four questions in order. Stop at the first branch that fires.

Q1: Does the step touch a secret, private dataset, or regulated record?
    yes -> Leaf D (split the step) unless you administer a dedicated host
Q2: Does the step need sustained CPU, memory, or disk (builds, long suites)?
    yes -> Leaf B (free server route) or Leaf C if you need an SLO
Q3: Is the step short-turn, low-egress, and non-recursive?
    yes -> Leaf A (hosted free model route)
Q4: Is the token spend unbounded or self-referential (loop can spawn loops)?
    yes -> Leaf D (downgrade to a script or cap the fan-out)

Step 1 — Write the task descriptor first

Do not classify in your head. Write the descriptor down, because it forces you to commit to numbers you can later falsify.

{
  "step_id": "nightly-docs-sync",
  "turns": 40,
  "egress": ["git"],
  "touches_secrets": false,
  "cpu_minutes": 3,
  "token_budget": 200000,
  "recursive": false,
  "latency_slo_s": null
}

The descriptor is the input to a classifier you can unit test. That is the whole trick: placement stops being an opinion once it is a pure function.

Step 2 — Classify with a pure function

# route.py — proposal, not production code. Extend the rules for your own risk model.
LEAVES = {
    "A": "hosted-free-model-route",
    "B": "free-server-route",
    "C": "administered-host-route",
    "D": "split-or-script",
}

def route(task: dict) -> str:
    if task.get("touches_secrets") and not task.get("administered_host"):
        return LEAVES["D"]          # Q1 fires
    if task.get("cpu_minutes", 0) >= 10:
        return LEAVES["B"]          # Q2 fires, cheap host for heavy work
    if task.get("latency_slo_s") is not None:
        return LEAVES["C"]          # SLOs and free hosting rarely coexist
    if task.get("recursive") or task.get("token_budget", 0) > 1_000_000:
        return LEAVES["D"]          # Q4 fires
    return LEAVES["A"]

Run it against the descriptor from Step 1. cpu_minutes=3, recursive=false, no SLO, no secrets, so it returns Leaf A. Now the interesting part: each leaf needs a probe, or you are guessing again.

Leaf A — hosted free model route

The failure mode here is not cost. It is cold start and silent budget drift across many short turns. Measure time to first response before you convert a script into a loop.

# Placeholders: confirm env var names and endpoint shape in the project docs.
start=$(date +%s%3N)
curl -sS -o /tmp/first.json -w 'http=%{http_code}\n' \
  -H "Authorization: Bearer $MC_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"reply with the single word: ok"}]}' \
  "$MC_BASE_URL"
end=$(date +%s%3N)
echo "cold_start_ms=$((end-start))"

If cold start is a meaningful fraction of your per-turn latency, batch the turns. Forty one-shot calls is worse than four batched calls on every axis that matters.

Leaf B — free server route

Heavy steps want a host you can lease and inspect, not a request window. The probe is boring on purpose: confirm the machine before you trust it with a loop.

#!/usr/bin/env bash
set -euo pipefail
nproc; free -m | awk 'NR==2{print "mem_mb=" $2}'; df -h / | awk 'NR==2{print "avail=" $4}'
python3 -V; git --version
# Fail fast if egress you did not plan for is required.
curl -sS -m 5 -o /dev/null -w 'egress_pkg=%{http_code}\n' https://pypi.org/simple/ || echo 'egress_pkg=blocked'

Record the output in the route ledger. When a run regresses next month, the ledger tells you whether the host changed or the task did.

Leaf C — administered host route

You land here when you have an SLO, a compliance obligation, or secret residency you cannot delegate. The cost is real, and so is the control. Choose this leaf deliberately, and write down the reason in the ledger, because it is the leaf you will be tempted to abandon later for convenience.

Leaf D — split or script

This is the leaf people skip. A step that touches secrets or spawns unbounded sub-loops should not be one agent task at all. Split it: a deterministic script handles the privileged part, and the agent handles only the parts where judgment is the point. The glossary term that matters here is blast radius, and the cheapest way to shrink it is to remove the step from the loop.

A reproducible test plan

The classifier is testable without any network access. Five assertions are enough to catch the common regressions:

import route

CASES = [
    ({"touches_secrets": True}, "split-or-script"),
    ({"cpu_minutes": 30}, "free-server-route"),
    ({"latency_slo_s": 2}, "administered-host-route"),
    ({"recursive": True}, "split-or-script"),
    ({"turns": 5}, "hosted-free-model-route"),
]

for task, expected in CASES:
    got = route.route(task)
    assert got == expected, f"{task} -> {got}, expected {expected}"
print("route decisions ok")

Then keep a ledger CSV with four columns — run_id, route, budget_spent, outcome — and review it weekly. Placement arguments end when the ledger has rows in it.

Limitations and who should skip this

• Free model access and free server options are operator-supplied availability features. They are not SLAs, and free allocations change without notice.
• The tree is a triage tool, not a security control. A secret-touching step needs a real review, not a branch that reroutes it.
• If you need audited residency, contractual uptime, or predictable per-request cost, go straight to Leaf C and skip the free routes entirely.
• If your steps are all short, non-privileged, and non-recursive, you may not need a tree at all — one route and a ledger will do.

What to do next

Write the descriptor for your slowest agent step, run the classifier, and probe the leaf it lands on. I keep the whole thing in about sixty lines of Python and one shell script, which is roughly the cost of one confusing incident. If you want to try the hosted route without provisioning anything first, MonkeyCode's free model access and free server option are the operator-stated entry points — check the current terms, then verify them with the probes above rather than trusting a blog post, including this one.