Split Placement: Deciding Which Agent Steps Run Locally, on a Free Server, or on a Free Model Endpoint

02:10, nineteen steps done, thirty-one to go

Picture a nightly triage job. It has sixty steps, it starts at 02:00, and at 02:10 it has finished nineteen of them. Step twenty needs to pull text out of a PDF, and the function that calls the model endpoint has just started returning 429 on every third request.

The job does not fail. It stalls, half-done, with no record of which leaf of the pipeline was running when the wall appeared. (The scenario is illustrative; the numbers describe the shape of the problem, not a benchmark I ran.)

Most writing about free AI tiers answers one question: which provider should the whole job use. That question is too coarse. Inside a single job, different steps belong in different places, and the placement decision is the one you can actually test.

Why placement is a separate question from tier choice

A job card is a promise about a workload. A placement decision is a predicate about one step: does it need a filesystem, does it touch regulated data, is it retry-safe, how tight is its deadline. Two steps in the same job can have identical inputs and still belong on opposite ends of your infrastructure.

Two themes keep showing up in current AI coverage: features moving into the client, and fatigue with generated output. Both push the same direction. Compute that can stay local should stay local, and anything that leaves the machine should leave under an explicit rule rather than a default.

The free tier used as the concrete case below is MonkeyCode's, which its operator describes as including free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I have not independently verified the quota figure the operator quotes (described as 10M tokens) or the server specifications, so the harness below treats any quota as unknown and measures consumption instead of assuming headroom.

Glossary: ten terms this tree depends on

Placement leaf — the destination a step is assigned to. This article uses four: LOCAL, FREE_SERVER, FREE_MODEL, ESCALATE.

Data class — the egress sensitivity of a payload. I use three: PUBLIC, INTERNAL, REGULATED.

Deterministic step — a step whose output depends only on its input, so running it twice costs time and nothing else.

Retry-safe (idempotent) — a step that can be re-executed after a transport failure without duplicating a side effect.

p95_seconds — the measured 95th-percentile duration of a step on your hardware. Guessed values make the whole tree decorative.

Quota cliff — the moment a free allowance stops absorbing requests, after which every call is either throttled or billed.

Fallback ladder — the ordered list of endpoints a step may attempt before it escalates.

Outcome class — how a failed attempt should be interpreted: RETRYABLE, FATAL, or OK. Conflating the first two is the most common cause of hot retry loops.

Fixture hash — a short digest over the exact inputs a test used, recorded so a later run can prove it saw the same bytes.

Ledger row — one append-only record per attempt: step, endpoint, attempt index, outcome, elapsed seconds.

The tree: ordered predicates, first match wins

place(step)
|
+-- 1. data_class == REGULATED? -----------------> LEAF D: ESCALATE
+-- 2. deterministic AND p95 <= 2s AND no disk? --> LEAF A: LOCAL
+-- 3. needs_disk OR not text_only? --------------> LEAF B: FREE_SERVER
+-- 4. INTERNAL AND not retry_safe? --------------> LEAF D: ESCALATE
+-- 5. p95 > 0.5 * deadline? ---------------------> LEAF D: ESCALATE
+-- 6. otherwise (text, retry-safe) --------------> LEAF C: FREE_MODEL

The order is the contract, not a style choice. Move predicate 1 below predicate 6 and you have an egress incident with a passing test suite.

The four leaves, with a worked example each

Leaf A: LOCAL

Worked example: validate 12,000 issue bodies against a JSON Schema before anything else runs. The step is pure, takes under two seconds at p95, and never needs the network.

# placement.py - pure decision logic, no network, no API keys
from dataclasses import dataclass
from enum import Enum


class Leaf(str, Enum):
    LOCAL = "local"
    FREE_SERVER = "free_server"
    FREE_MODEL = "free_model"
    ESCALATE = "escalate"


class DataClass(str, Enum):
    PUBLIC = "public"        # safe to send to a third-party endpoint
    INTERNAL = "internal"    # company-internal, not customer data
    REGULATED = "regulated"  # PII, health, payment, or contract-bound


@dataclass(frozen=True)
class Step:
    name: str
    p95_seconds: float      # measured on your machine, not guessed
    deterministic: bool
    text_only: bool
    needs_disk: bool
    data_class: DataClass
    retry_safe: bool
    deadline_seconds: float | None = None


def place(step: Step) -> tuple[Leaf, str]:
    """Ordered predicates. First match wins, so order is the contract."""
    if step.data_class is DataClass.REGULATED:
        return Leaf.ESCALATE, "regulated payload: no third-party egress"

    if step.deterministic and not step.needs_disk and step.p95_seconds <= 2.0:
        return Leaf.LOCAL, "cheap pure function: egress would be waste"

    if step.needs_disk or not step.text_only:
        return Leaf.FREE_SERVER, "needs a filesystem or a non-text runtime"

    if step.data_class is DataClass.INTERNAL and not step.retry_safe:
        return Leaf.ESCALATE, "non-idempotent internal call on a shared endpoint"

    if step.deadline_seconds and step.p95_seconds > step.deadline_seconds * 0.5:
        return Leaf.ESCALATE, "free-tier latency variance can eat the deadline"

    return Leaf.FREE_MODEL, "stateless text transform, retry-safe"

Leaf B: FREE_SERVER

Worked example: extract text from 400 PDFs and keep an on-disk cache so a second run is nearly free. This step needs a filesystem and CPU time, not a model.

This is also the leaf where a free server option changes the cost shape. Compute-heavy, non-text steps have no reason to touch a model endpoint at all, and putting them there wastes the resource you are actually short on.

Leaf C: FREE_MODEL

Worked example: classify and summarize issue text, one call per item, no local state. Text in, text out, retry-safe, egress class PUBLIC.

Before this leaf is reachable you need a reason to believe the endpoint tolerates your request rate. That is what the ladder and the ledger below are for.

Leaf D: ESCALATE

Worked example: redact personal data before any summarization happens. The step is a queue write plus a human or in-boundary worker, never a third-party call.

Escalation is a leaf, not a failure. If your tree has no ESCALATE, it is not a decision tree; it is a preference list.

A reproducible harness in five steps

1. Pin the fixture hash. Record the exact bytes a run consumed, so a later result can be tied to a specific input set.

from hashlib import sha256

def fixture_hash(paths):
    h = sha256()
    for p in sorted(paths):
        h.update(p.encode())
        h.update(open(p, "rb").read())
    return h.hexdigest()[:12]

2. Pin the case table. Every leaf needs at least one case, including the ones you expect to be rare.

CASES = [
    (Step("parse_json", 0.3, True, True, False, DataClass.PUBLIC, True), Leaf.LOCAL),
    (Step("extract_pdf", 12.0, False, False, True, DataClass.INTERNAL, True), Leaf.FREE_SERVER),
    (Step("summarize_issue", 3.1, False, True, False, DataClass.PUBLIC, True), Leaf.FREE_MODEL),
    (Step("redact_pii", 4.0, False, True, False, DataClass.REGULATED, True), Leaf.ESCALATE),
]

3. Separate retryable from fatal. A 429 and a schema mismatch must not share a code path.

class Outcome(str, Enum):
    OK = "ok"
    RETRYABLE = "retryable"  # 429, 503, timeout
    FATAL = "fatal"          # 400, 401, schema mismatch


def run_ladder(step, endpoints, call, ledger):
    for attempt, endpoint in enumerate(endpoints, start=1):
        try:
            result = call(endpoint, step)
        except Retryable as exc:
            ledger(step.name, endpoint, attempt, Outcome.RETRYABLE, exc)
            continue
        except Fatal as exc:
            ledger(step.name, endpoint, attempt, Outcome.FATAL, exc)
            return Leaf.ESCALATE, exc
        ledger(step.name, endpoint, attempt, Outcome.OK, None)
        return endpoint, result
    return Leaf.ESCALATE, "ladder exhausted"

4. Prove the ladder with a stub, not with a live quota. A stub that fails on the first endpoint is deterministic, free, and runs in CI.

def test_ladder_falls_through():
    def call(endpoint, step):
        if endpoint == "model-a":
            raise Retryable("429")
        return "ok"

    endpoint, result = run_ladder(step, ["model-a", "free-server"], call, ledger=lambda *a: None)
    assert endpoint == "free-server"
    assert result == "ok"

5. Write a ledger row per attempt. Append-only JSONL is enough, and it is the only artifact that tells you where a quota went when it disappears.

import json, time

def ledger(step_name, endpoint, attempt, outcome, error, path="ledger.jsonl", t0=None):
    row = {
        "step": step_name,
        "endpoint": endpoint,
        "attempt": attempt,
        "outcome": outcome.value,
        "error": type(error).__name__ if error else None,
        "seconds": round(time.monotonic() - t0, 3) if t0 else None,
    }
    with open(path, "a") as fh:
        fh.write(json.dumps(row) + "\n")

Run the whole thing with pytest -q. If you want the leaf distribution for a real job, feed your step table through place() and count the leaves before you deploy anything.

Limitations and what I did not verify

Free tiers move. A tree that encodes this month's allowances will quietly lie next quarter, which is why the harness measures instead of assuming headroom. The p95_seconds values must come from your own runs; copy-pasting mine would produce confident nonsense.

The data-class verdicts are jurisdictional and contractual. REGULATED here means "handled under a reviewed boundary" and nothing more, so the compliance owner decides those rows, not the engineer writing the predicate.

The ladder costs latency. Each retryable failure adds a round trip, and any non-idempotent step placed on FREE_MODEL by mistake can duplicate a side effect instead of failing cleanly. Retry-safety is a property of your step, not of the endpoint.

The described MonkeyCode free tier and its reported quota are operator-supplied statements that I have not independently tested against live rate limits, model behavior, or uptime.

Who should not use this

Teams with hard contractual SLAs should not route deadline-critical steps through any free tier, whatever the tree says. Workloads dominated by large non-text payloads will spend more time uploading than computing and should keep those steps local or on a server you control. Anyone without an owner for the REGULATED branch should stop at the glossary and get one first.

If you have a nightly job with a mixed step table, run place() over it and post the leaf counts. The distribution usually says more about where your budget goes than the tree does, and it is a cheap thing to compare across two teams.