Letter to Pre-CI-Me: Make the Token Ledger Fail the Build

Three weeks ago I edited one sentence in a system prompt. It read cleaner. It also doubled the steps the agent took per task. No test failed. No alarm fired. The allowance drained quietly over two days.

I keep writing these letters to the version of me who starts the next harness. This one covers the gap I keep falling into. I measured agent cost. I never enforced it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What Went Wrong, in Order

1. A ledger nobody reads. I logged tokens per run into a file. The file grew. Nothing compared today to last week.
2. Tests against the live model. Every check called a real endpoint. Same prompt, different step count, no way to tell drift from noise.
3. No per-task ceiling. One tool-loop bug ran 41 steps before I noticed. The loop was legal, so nothing stopped it.

All three have the same fix. Record a run once, commit it as a fixture, then fail the build when the next run drifts.

Step 1 — Record Once, Replay Forever

The gate must never touch the network. That keeps it fast, deterministic, and free. You record fixtures on a real model, then commit the numbers.

// recorder.mjs — sketch. I have not executed this exact file.
// Run once per case against a live model, then commit the JSON it prints.
import { writeFileSync } from "node:fs";

const CASES = ["summarize-issue", "rename-symbol"];
const out = { version: 1, cases: {} };

for (const name of CASES) {
  const run = await runAgentCase(name); // your existing harness call
  out.cases[name] = { steps: run.steps, tokens: run.promptTokens + run.completionTokens };
}

writeFileSync("run.json", JSON.stringify(out, null, 2));

Two fixtures matter: a baseline you trust and a current run. Keep them in the same shape or the gate cannot diff them.

Step 2 — Commit the Baseline

{
  "version": 1,
  "cases": {
    "summarize-issue": { "steps": 4, "tokens": 1500 },
    "rename-symbol": { "steps": 2, "tokens": 640 }
  }
}

Commit baseline.json with the same care as a lockfile. Update it in the same pull request that intentionally changes prompt or tool schema. Never update it in a fix-up commit.

Step 3 — Write the Gate

Zero dependencies. Node 18 and up.

// gate.mjs
import { readFileSync } from "node:fs";

const load = (p) => JSON.parse(readFileSync(p, "utf8"));
const baseline = load(process.argv[2] ?? "baseline.json");
const current = load(process.argv[3] ?? "run.json");

const TOKEN_GROWTH_LIMIT = 1.2; // allow 20% drift
const STEP_GROWTH_LIMIT = 1;    // allow one extra step

const failures = [];
for (const [name, base] of Object.entries(baseline.cases)) {
  const now = current.cases[name];
  if (!now) { failures.push(`${name}: missing from run`); continue; }
  if (now.tokens > Math.ceil(base.tokens * TOKEN_GROWTH_LIMIT)) {
    failures.push(`${name}: tokens ${base.tokens} -> ${now.tokens}`);
  }
  if (now.steps > base.steps + STEP_GROWTH_LIMIT) {
    failures.push(`${name}: steps ${base.steps} -> ${now.steps}`);
  }
}

if (failures.length) {
  console.error("cost gate failed:\n" + failures.map((f) => " - " + f).join("\n"));
  process.exit(1);
}
console.log(`cost gate passed for ${Object.keys(baseline.cases).length} cases`);

Run it locally before you trust it:

node gate.mjs baseline.json run.json
# exit 0 when clean, exit 1 with a per-case diff when it drifts

The thresholds are the whole argument. Twenty percent absorbs normal sampling variance. One extra step catches tool-loop regressions without flagging every model update.

Step 4 — Wire It Into CI

name: cost-gate
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: node gate.mjs baseline.json run.json

Now a prompt edit that doubles steps fails review. That is the only change I actually needed three weeks ago.

Step 5 — Record on a Disposable Box

The recording step is the only part that costs quota or needs a machine. That is where I now use MonkeyCode's free model access and free server option: record once on a throwaway box, commit the fixtures, delete the box. Check the project page for the current allowance and limits before you plan around them, since availability can change.

Open-source MonkeyCode is worth a look if you want to run this harness without a paid key on day one. The gate itself stays provider-neutral. That is deliberate, because the fixture format outlives any single endpoint.

When This Gate Pays Off

Situation	Gate here?	Reason
Prompt or tool schema changes weekly	Yes	Drift shows up as extra steps
Shared team allowance	Yes	One pull request can drain it
One-off script, run twice	No	Baseline upkeep exceeds savings
Measuring latency or answer quality	No	This gate counts steps and tokens only

Limits and Who Should Skip It

This gate is a regression alarm, not a cost model. Fixtures go stale when the upstream model changes behavior. Token counts from a recorder are estimates of your own pipeline, not provider billing. And it says nothing about whether the output is still correct, so keep your assertion tests beside it.

Skip it if you run fewer than a dozen agent calls a day. Skip it if prompts live in a spreadsheet nobody versions. Skip it if you need latency percentiles, because a replay with no network cannot give you those.

What I Would Tell Myself

Do not build another dashboard. Build one file that exits non-zero. The ledger was never the useful artifact. The refusal was.

If you want to try the recording step without setup overhead, the MonkeyCode project is a reasonable place to start, and the gate above works the same either way.