Letter to Pre-Router-Me: Route the Job Before You Spend a Token

Last week I lost a day to one missing decision.

Every job I owned ran on the same box. A README summarizer, a repo scanner, and a stateful indexer shared one default.

The box restarted mid-afternoon. The summarizer recovered fine. The indexer came back empty.

I spent hours blaming the model. It was not a model problem. It was a routing problem.

This letter is what I should have written before the first run. Three mistakes, one router, one test plan.

Mistake 1: Treating a disposable box as a worker

A free server is disposable by design. That is the feature, not a bug.

Anything that must survive a restart does not belong there. Counters, caches, queues, and lock files all reset.

Verify this before you trust any box. One marker test is enough.

touch /tmp/marker && echo "marker set"
# restart or redeploy the server here
test -f /tmp/marker && echo "state survived" || echo "state gone"

If the marker is gone, no job on that box may own state. Keep the box for stateless steps only.

Mistake 2: Routing by size instead of shape

Big tasks feel like they need big machines. That instinct is usually wrong.

Shape beats size for routing decisions. Shape means three questions: does it write state, does it need network egress, does it execute code.

A 200-token job that writes a shared database is riskier than a 5,000-token summarizer. Size gave me nothing. Shape gave me a rule.

Here is the rule I use now.

Job shape	Target	Why
Writes shared state	Local or managed store	Ephemeral boxes lose it
Writes local state, single run	Local	A restart kills the run
Executes untrusted code, no state	Disposable server	Cheap to throw away
Read-only reasoning, short output	Free model access	No state, easy to retry
Read-only, long output, flaky upstream	Local	Retries get expensive

Four rows would have saved my afternoon. The table is not clever. It is just written down.

Mistake 3: No stop condition

I retried one failing step for most of an evening. Same input, same prompt, same error.

Retries without a ceiling are a budget leak. The counter has to be per step name, not per run.

Two numbers are enough. Max three attempts per step name, max one retry per distinct error string.

After that, stop and write a diff. Silent loops are worse than loud failures.

The artifact: a router that runs before any model call

This is a 40-line classifier. Standard library only. It reads a job description on stdin and prints a target.

#!/usr/bin/env python3
# route.py - decide the target before spending anything
import json, sys
from dataclasses import dataclass

@dataclass
class Job:
    name: str
    executes_code: bool
    writes_state: bool
    needs_egress: bool
    max_out_tokens: int
    blast_radius: str  # read-only | writes-local | writes-shared

def route(job):
    if job.blast_radius == 'writes-shared':
        return 'local', 'shared writes never leave your machine'
    if job.writes_state:
        return 'local', 'ephemeral boxes drop state on restart'
    if job.executes_code and job.blast_radius == 'read-only':
        return 'free-server', 'disposable execution, nothing to keep'
    if job.blast_radius == 'read-only' and job.max_out_tokens <= 2000:
        return 'free-model', 'short read-only reasoning'
    return 'local', 'long output or egress makes retries costly'

if __name__ == '__main__':
    job = Job(**json.load(sys.stdin))
    target, why = route(job)
    print(json.dumps({'job': job.name, 'target': target, 'reason': why}))

Feed it a job and read the verdict.

echo '{"name":"summarize-readme","executes_code":false,"writes_state":false,"needs_egress":false,"max_out_tokens":600,"blast_radius":"read-only"}' | python3 route.py

Expected output:

{"job": "summarize-readme", "target": "free-model", "reason": "short read-only reasoning"}

Now flip one field. Set "writes_state": true and run it again. The target must move to local.

That flip is the regression test. It costs one second and catches the whole bug class I hit.

Step budget: a wrapper with a hard ceiling

Routing decides where a job runs. It does not stop a loop.

Add a wrapper that caps each step and logs every exit code. Separate streams, one JSONL ledger.

#!/usr/bin/env bash
# guard.sh - one step, one ceiling, one ledger line
set -uo pipefail
STEP="${1:?step name}"; LIMIT="${2:-120}"; shift 2
mkdir -p out
start=$(date +%s)
timeout "$LIMIT" "$@" >"out/$STEP.stdout" 2>"out/$STEP.stderr"
rc=$?
printf '{"step":"%s","rc":%d,"secs":%d}\n' "$STEP" "$rc" "$(( $(date +%s) - start ))" >> ledger.jsonl
exit $rc

Then read the ledger, not the transcript.

python3 - <<'PY'
import json, collections
rows = [json.loads(l) for l in open('ledger.jsonl')]
bad = [r for r in rows if r['rc'] != 0]
print('steps', len(rows), 'failed', len(bad))
print(collections.Counter(r['step'] for r in bad).most_common(3))
PY

The counter answers one question fast. Which step name repeats?

If one name dominates the failures, stop. The bug is in that step, not in the model.

A test plan you can finish in ten minutes

1. Run the marker test on the box. Record the result.
2. Classify five real jobs with route.py. Write down each target.
3. Set writes_state: true on all five. Confirm every target moves to local.
4. Wrap two steps in guard.sh with a 60-second ceiling. Confirm the ledger has two lines.
5. Force a failure with false. Confirm the counter reports exactly one hit.

Five steps, zero model calls. That is the point.

Where free access actually fits

MonkeyCode offers free model access and a free server option, per the operator. I use both in narrow slots.

Free model access handles short, read-only tasks in my router. The free server handles throwaway execution with no state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quota terms and hardware change. Check the current product page before planning a budget around them. I deliberately left numbers out here, because stale numbers are worse than none.

Limitations

The router is a heuristic. It cannot tell you whether a task is correct.

Latency varies by time of day. A passing run tonight does not prove a passing run tomorrow.

The marker test checks one restart. Long-lived processes may behave differently.

Nothing here covers secrets. Keep credentials off shared boxes entirely.

Who should not use this

Skip it if your jobs write shared state. Move those to a managed store first.

Skip it if you handle regulated data. The data-flow review comes before any free tier.

Skip it if you need hard latency guarantees. A discretionary free box is not a scheduler.

What I would tell myself now

Write the routing table before the first job. Two minutes, five rows, done.

Then add one ledger line per step. Read the counter, not the transcript.

If you want to test the router, start with the read-only fixtures. They are the cheapest place to be wrong.