Last Tuesday's pairing hour started with a staging config and ended with a 90-line script that runs before the session, not during it. A senior engineer on the team opened the repo, read the AGENTS.md we had been growing for three weeks, and asked a question that reframed the whole hour: "What in this file is actually enforced?"
Nothing was. It was a policy doc for a coding agent — retry rules, model preferences, a polite note about not touching /v2 endpoints — and it had the same failure mode as every policy doc: it described intent and relied on the model to comply.
This is the record of that hour: the three questions, the three dead ends, and the one artifact we kept.
The three questions
The senior did not propose a solution for the first twenty minutes. She asked:
- How do you know the endpoint is reachable right now, as opposed to reachable when you last checked?
- What happens on a cold start when the free server option has not served a request in a while?
- Which number in your ledger would you defend if someone asked you to prove it?
Question three did the damage. We had a token ledger from an earlier session and a wall-clock number we had been quoting in standups. Neither had a timestamp or a request shape attached, so neither was defensible. They were vibes with units.
Dead end 1: a longer policy doc
First instinct, and the cheapest to try: add a section to AGENTS.md titled "Retry and fallback protocol." We wrote six bullets — try once, back off, do not retry on schema errors, prefer the primary model, log the attempt.
The agent read the file and ignored the bullets on the first failing request. That is not a model defect. A policy doc is input to a sampling process, not a control flow construct. If a rule has to hold every time, it belongs in a script or a gateway, not in prose.
We deleted the section. The file got shorter and more honest.
Dead end 2: retry logic inside the prompt wrapper
Next attempt: put the retry loop in the wrapper that assembles prompts. That gave us two loops — one in the wrapper, one in the client code that called the wrapper — and it took a full hour to notice, because the symptom was just "sometimes slow."
Double retries are expensive in a way that is hard to see. A single failure becomes four requests, the ledger shows one logical call, and the latency distribution gets a fat tail nobody can explain. We had already been bitten by this pattern once in a much smaller form.
Rule we kept from it: a retry loop must own a single, named layer, and that layer must write its own ledger row per attempt.
Dead end 3: a queue in front of the model
Third idea, and the one the senior vetoed fastest: park requests in a queue with exponential backoff so slow responses stop being visible.
Her objection was not performance. It was observability. A queue converts "the endpoint is struggling" into "the queue is deep," which is a strictly worse signal during a pairing hour, because the pairing hour is the moment you least want to debug your own infrastructure. Queues are for throughput. We did not have a throughput problem. We had a can I trust this endpoint right now problem.
What we kept: a preflight probe with a fallback ladder
The artifact is a single script that runs before the session, writes one JSON line per probe to a ledger file, and prints a verdict: proceed, degrade, or stop. It never retries during the session. It decides once, up front, and the session inherits the decision.
This is where MonkeyCode's free model access and free server option became part of the method rather than a footnote. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The point of a preflight probe is that you can afford to measure before you commit; a free tier you can probe without a card is a good place to run the ladder the first time, and the free server option makes cold-start behavior something you can actually observe instead of guess at. Availability, current quotas, and terms change, so check the official docs rather than trusting a number in a blog post — including this one.
The probe
// preflight.mjs — run before a pairing session, not during one
import { appendFile } from "node:fs/promises";
const LEDGER = new URL("./preflight-ledger.jsonl", import.meta.url);
// Endpoints come from env. Do not hardcode model names in the script;
// the ladder is data, and data changes faster than code.
const LADDER = [
{ id: "primary", url: process.env.MC_URL_A, model: process.env.MC_MODEL_A },
{ id: "fallback", url: process.env.MC_URL_B, model: process.env.MC_MODEL_B },
];
async function record(row) {
await appendFile(LEDGER, JSON.stringify({ at: new Date().toISOString(), ...row }) + "\n");
}
// Probe 1: reachability + time to first byte, with a hard deadline.
async function probeReach(ep, timeoutMs = 15000) {
const ac = new AbortController();
const t = setTimeout(() => ac.abort(), timeoutMs);
const started = performance.now();
try {
const res = await fetch(`${ep.url}/models`, { signal: ac.signal });
return { ok: res.ok, status: res.status, ttfbMs: Math.round(performance.now() - started) };
} catch (err) {
return { ok: false, status: 0, error: String(err.name) };
} finally {
clearTimeout(t);
}
}
// Probe 2: does the endpoint return a tool call whose arguments parse?
// Shape only. It cannot tell you whether the chosen tool was the right one.
async function probeToolShape(ep, timeoutMs = 20000) {
const ac = new AbortController();
const t = setTimeout(() => ac.abort(), timeoutMs);
const started = performance.now();
try {
const res = await fetch(`${ep.url}/chat/completions`, {
method: "POST",
signal: ac.signal,
headers: {
"content-type": "application/json",
authorization: `Bearer ${process.env.MC_KEY}`,
},
body: JSON.stringify({
model: ep.model,
messages: [{ role: "user", content: "Call the ping tool with id 1." }],
tools: [{
type: "function",
function: {
name: "ping",
parameters: { type: "object", properties: { id: { type: "number" } }, required: ["id"] },
},
}],
}),
});
const body = await res.json();
const call = body?.choices?.[0]?.message?.tool_calls?.[0];
let argsOk = false;
try { argsOk = typeof JSON.parse(call.function.arguments).id === "number"; } catch {}
return { ok: Boolean(call) && argsOk, ttftMs: Math.round(performance.now() - started) };
} catch (err) {
return { ok: false, error: String(err.name) };
} finally {
clearTimeout(t);
}
}
const verdicts = [];
for (const ep of LADDER.filter((e) => e.url && e.model)) {
const reach = await probeReach(ep);
await record({ probe: "reach", endpoint: ep.id, ...reach });
if (!reach.ok) { verdicts.push({ id: ep.id, verdict: "stop" }); continue; }
const shape = await probeToolShape(ep);
await record({ probe: "tool_shape", endpoint: ep.id, ...shape });
verdicts.push({ id: ep.id, verdict: shape.ok ? "proceed" : "degrade" });
}
const chosen = verdicts.find((v) => v.verdict === "proceed");
console.log(JSON.stringify({ chosen: chosen?.id ?? null, verdicts }, null, 2));
process.exit(chosen ? 0 : 1);
What the ledger rows look like
Values below are placeholders, not measurements. The shape is the artifact; the numbers are yours to fill on the day you run it.
{"at":"<iso>","probe":"reach","endpoint":"primary","ok":true,"status":200,"ttfbMs":"<measured>"}
{"at":"<iso>","probe":"tool_shape","endpoint":"primary","ok":true,"ttftMs":"<measured>"}
{"chosen":"primary","verdicts":[{"id":"primary","verdict":"proceed"}]}
Why a ladder and not a retry
A retry reacts to failure after the user is already waiting. A ladder picks the endpoint before the first request and records why. The difference shows up in the retro: with a ladder you can say "we ran on the fallback for two days" because the ledger has the rows. With a retry you get a latency graph and a shrug.
The decision table we kept next to it
| Situation | Preflight verdict | Action |
|---|---|---|
| Primary reachable, tool shape valid | proceed |
Run the session, no fallback |
| Primary reachable, tool shape invalid | degrade |
Run read-only work; no file writes |
| Primary unreachable, fallback valid | proceed |
Run on fallback, note it in the PR description |
| Both unreachable | stop |
Do not start. Pairing time is too expensive to spend on transport errors |
| Probe passes, three failures in ten calls | — | Stop trusting the probe for the day; it sampled a healthy moment |
That last row mattered more than any code. A probe is a snapshot. On a shared free tier, a healthy snapshot at 09:00 says very little about 14:00.
What this does not tell you
- Cold-start behavior under your real prompt size. The probe sends a trivial message. Context length changes the latency curve, and the probe will not model it.
- Semantic correctness. A tool call can parse and still be the wrong tool. Shape checks catch breakage, not bad judgment.
- Load. One sequential request per endpoint is not a throughput test. Do not quote these numbers as capacity.
- Anything about tomorrow. Endpoint names, limits, and terms on any provider change. Re-run the probe; do not trust the ledger from last month, including your own.
Who should not use this
Teams with a contractual latency SLA, workloads with data-residency constraints that a free tier cannot satisfy, and anyone who cannot commit a ledger file to version control. If your agent runs unattended overnight, a preflight check is necessary but not sufficient — you need alerting on the ledger, not a human reading it at breakfast.
The thing the senior said at the end
She closed her laptop and said the probe was worth keeping mainly because it made the absence of a decision visible. Before it, "which endpoint are we on today" had no answer, so nobody asked.
If you want to run the ladder against a free tier before wiring it to a paid one, the project's free model access and free server option are the low-stakes place to start — keep the ledger in the repo and let the JSON lines argue for you.
Tags: ai, programming, devtools, productivity.
Top comments (0)