The Free-Lane Admission Test: Four Checks Before an AI Feature Ships on a Free Model

Agent demos are everywhere this month. Most of them run on a free model lane somewhere.

Free lanes are not the problem. Shipping on one without an admission gate is.

This is a field guide for the opposite question. Not "how do I get on a free lane" but "what proves I should not be on one yet."

The Friday deploy that doubled every refund

A two-person team wired a triage agent to a free model lane. The model classified tickets and triggered refunds. Local tests passed. Nobody tested the retry path.

On Monday, finance found 41 duplicate refunds. The model had returned valid JSON both times the worker retried. It answered correctly. The integration had no gate.

That is the recurring shape of these incidents. The lane behaves. The caller absorbs the failure.

A paid lane with a contract puts a support engineer behind the failure. A free lane leaves it with whoever called it.

Where a free lane genuinely earns its place

Free model access is a real fit for some workloads. The pattern is consistent.

• Low-stakes classification where a wrong label costs a re-run, not money.
• Draft generation that a human edits before anything downstream reads it.
• Batch enrichment over data you can recompute cheaply.
• Prototypes where the goal is to learn whether the feature is worth building.

Four red flags that should stop the deploy

• The output directly triggers a money movement, deletion, or signed callback.
• The lane is asked to classify its own retryability instead of a deterministic rule doing it.
• The caller's timeout budget is smaller than the lane's tail latency.
• Business state must be inside the prompt for the answer to be correct.

Any single flag is enough to pause. Two flags means the design is wrong, not the lane.

A runnable admission test

The harness below replays frozen inputs against any lane and fails the build on three gates. It runs without network access against a local adapter. Swap one function to point it at a live endpoint.

// free-lane-admission.mjs
// Usage: node free-lane-admission.mjs fixtures/triage.jsonl
// Node 20+. No dependencies.
import { readFileSync } from "node:fs";

const REPLAY_PATH = process.argv[2] ?? "fixtures/triage.jsonl";
const GATE = { schemaRate: 0.995, p99Ms: 2500, effectKeyRate: 1.0 };

const cases = readFileSync(REPLAY_PATH, "utf8")
  .split("\n")
  .filter((line) => line.trim().length > 0)
  .map((line) => JSON.parse(line));

// Swap this body for a live call. It runs twice per case on purpose.
async function callLane(input) {
  // Unrun live example:
  // const r = await fetch(LANE_URL, { method: "POST", body: JSON.stringify(input) });
  // return r.json();
  return { label: "billing", effect_key: `refund:${input.ticket_id}` };
}

const SCHEMA = {
  label: (v) => ["billing", "bug", "howto"].includes(v),
  effect_key: (v) => typeof v === "string" && v.length > 0,
};

function schemaOk(reply) {
  return Object.entries(SCHEMA).every(([key, test]) => test(reply?.[key]));
}

function p99(samples) {
  const sorted = [...samples].sort((a, b) => a - b);
  const idx = Math.ceil(sorted.length * 0.99) - 1;
  return sorted[Math.max(0, idx)];
}

const latencies = [];
let schemaPass = 0;
let keyStable = 0;

for (const testCase of cases) {
  const t0 = performance.now();
  const first = await callLane(testCase.input);
  const second = await callLane(testCase.input);
  latencies.push(performance.now() - t0);

  if (schemaOk(first) && schemaOk(second)) schemaPass += 1;
  if (first.effect_key === second.effect_key) keyStable += 1;
}

const total = cases.length;
const result = {
  cases: total,
  schema_rate: schemaPass / total,
  effect_key_rate: keyStable / total,
  p99_ms: p99(latencies),
};

const failures = [];
if (result.schema_rate < GATE.schemaRate) failures.push("schema_rate");
if (result.effect_key_rate < GATE.effectKeyRate) failures.push("effect_key_rate");
if (result.p99_ms > GATE.p99Ms) failures.push("p99_ms");

console.table(result);
if (failures.length > 0) {
  console.error("BLOCKED on: " + failures.join(", "));
  process.exit(1);
}
console.log("ADMITTED: lane may carry this workload behind a queue.");

Exit code 1 is the useful part. It belongs in CI next to the unit tests.

What each gate is actually measuring

1. schema_rate catches format drift before it reaches a parser. A free lane can change behavior between deploys without notice.
2. effect_key_rate catches non-idempotent side effects. Calling the lane twice with the same input must produce the same effect key.
3. p99_ms catches the timeout trap. A lane that is fast on average and slow at the tail will exhaust retries during an incident.

What stays out of the prompt

Retry classification must be a rule table, not a model decision. The mapping below is boring and correct.

Transport signal	Classification	Caller action
Connection reset before headers	Retryable	Retry with jitter, cap 3
HTTP 429	Retryable	Back off, respect Retry-After
HTTP 400 with schema error	Non-retryable	Dead-letter, alert
Timeout after request sent	Unknown	Do not blind-retry writes

Exit criteria: when to leave the free lane

Adoption without exit criteria is how a prototype becomes an outage. Write these down before launch.

• Schema conformance falls below 99.5% on two consecutive days.
• p99 latency exceeds half the caller's timeout budget.
• Any duplicate effect key appears in production logs.
• Token spend per request grows threefold with no feature change.
• The workload starts needing business state inside the prompt.

When a threshold trips, the migration is usually one of four moves. Move binary decisions to deterministic rules. Move high-volume low-variance work to a self-hosted small model. Move anything with an SLA to a paid lane with a contract. Move repeated inputs to a cache keyed on normalized input.

Who should not use this approach

• Teams without a frozen replay set. The harness measures nothing without one.
• Products where a wrong classification is a compliance event.
• Workloads whose inputs change shape weekly. Fixtures rot faster than the gate pays off.
• Anyone who cannot run a queue in front of the lane.

Limits of the test

The harness measures conformance, idempotency, and tail latency. It does not measure answer quality.

A lane can pass every gate and still produce wrong labels. Quality needs a labeled sample and a human review loop. That is a separate artifact, and it does not fit in a CI job.

Free lanes also change. Terms, rate limits, and model routing move without a changelog. The admission test is a snapshot, not a permanent clearance. Re-run it after any lane-side change.

MonkeyCode in this workflow

The operator describes MonkeyCode as an open-source project that offers free model access, a free token allowance, and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Those claims are operator-supplied. Confirm current limits on the project's own page before sizing anything against them.

The admission test above is deliberately lane-agnostic. It is the same gate whether the lane is free, self-hosted, or paid. Run it on a free lane to find out what the lane cannot carry yet, not to prove it works.

That inversion is the point. If the harness reports ADMITTED, the free lane has earned a spot behind a queue. If it reports BLOCKED, the harness just saved a weekend.

Readers who want to try the free lane can point the callLane body at it and run the fixtures they already have.