Cost Regression Should Fail the Build

You will not notice a cost regression by reading an invoice. Invoices arrive after the traffic, after the incident review, and after the person who wrote the expensive loop has moved to another team. You need the build to tell you, on the commit that caused it, with a number and a threshold and someone's name on it.

That means treating cost per successful outcome as a test assertion. Same as a snapshot test, same as a schema check: a fixture, a threshold, a deterministic runner, and a red build when it moves.

The usual objection is that cost tests are expensive to run. They are — if you point them at a live endpoint. So don't measure live. Record once, replay forever, and put the gate behind the replay.

One run, five numbers

Start with a frozen record of a single run. Five fields matter, and wall clock is two of them.

# costgate/model.py
from dataclasses import dataclass
from fractions import Fraction

@dataclass(frozen=True)
class Run:
    id: str
    usd_model: Fraction     # metered model spend for this run
    usd_infra: Fraction     # server minutes, billed or amortized
    tokens_in: int
    tokens_out: int
    retries: int
    wall_ms: int
    queued_ms: int
    ok: bool

    @property
    def spend(self) -> Fraction:
        return self.usd_model + self.usd_infra

Two things matter here. Money is a Fraction, not a float, because a gate that compares 0.041000000000000004 to 0.041 will fail on a Tuesday for no reason. And ok is present because the denominator of your metric is not "runs" — it is "runs that actually produced the thing you sold."

The divisor is the whole test

Dividing spend by successes is the part people skip, and skipping it is why retry storms look free. When a call fails twice and succeeds on the third attempt, the numerator grows by three attempts of spend while the denominator grows by one. Nothing else in your observability captures that shape.

# costgate/gate.py
from fractions import Fraction

def cost_per_success(runs):
    spend = sum((r.spend for r in runs), Fraction(0))
    wins = sum(1 for r in runs if r.ok)
    return None if wins == 0 else spend / wins

def retry_share(runs):
    attempts = sum(r.retries + 1 for r in runs)
    return Fraction(sum(r.retries for r in runs), attempts)

def p95(values):
    if not values:
        return 0
    s = sorted(values)
    return s[max(0, int(round(0.95 * (len(s) - 1))))]   # nearest-rank

Note that cost_per_success returns None instead of raising when nothing succeeded. A cassette that fails every call should fail the gate with a readable message, not a ZeroDivisionError buried in a traceback.

Wire it to a budget file

Keep thresholds in version control next to the code they constrain, in a format a reviewer can diff.

# costgate/budget.toml
[gate]
max_cost_per_success_usd = "0.041"
max_retry_share          = "0.15"
max_p95_wall_ms          = 24000

Fraction("0.041") parses that string exactly, which is the reason for the quotes. Then the check itself stays boring:

# costgate/check.py
def check(runs, budget):
    failures = []
    cps = cost_per_success(runs)
    if cps is None:
        failures.append("no successful runs: cost/success undefined")
    elif cps > Fraction(budget["max_cost_per_success_usd"]):
        failures.append(f"cost/success {float(cps):.4f} over budget")
    if retry_share(runs) > Fraction(budget["max_retry_share"]):
        failures.append("retry share regressed")
    if p95([r.wall_ms for r in runs]) > int(budget["max_p95_wall_ms"]):
        failures.append("p95 wall clock regressed")
    return failures

Run it from CI with one command and let the exit code do the talking.

python -m costgate check --cassette fixtures/nightly.jsonl --budget costgate/budget.toml

The replay transport is the trick

A cassette is a recorded JSONL file: one row per model call, holding a hash of the prompt and the response that came back. Replay asserts the hash before returning the response, so a prompt change shows up as cassette drift instead of a silently passing test.

# costgate/replay.py
import hashlib, json

def sha(text):
    return hashlib.sha256(text.encode()).hexdigest()

class Replay:
    """Deterministic stand-in for a model call, backed by a recorded cassette."""
    def __init__(self, path):
        with open(path, encoding="utf-8") as fh:
            self.rows = [json.loads(line) for line in fh if line.strip()]
        self.i = 0

    def call(self, prompt):
        row = self.rows[self.i]
        assert row["prompt_sha"] == sha(prompt), f"cassette drift at row {self.i}"
        self.i += 1
        return row["response"]

Now the gate is deterministic, runs in seconds, and costs nothing. Refresh the cassette on a schedule so drift is a decision someone made, not a surprise. This reference implementation is not executed against your stack — treat it as a shape to copy, not a library to depend on.

Calibrate before you enforce

The first week of this gate will be red, and that is calibration, not a bug. Take the median cost per success across your last twenty nightlies, add a margin you can defend, and only then make the gate blocking. Until then, run it in report mode and write down who owns each threshold.

This is also where a free model tier and a free server option genuinely participate in the method rather than decorate it. Running a replay gate needs almost no throughput and almost no money; what it needs is somewhere to run that isn't your metered build fleet, and a way to re-record cassettes without watching a spend graph while you wait. MonkeyCode's free model access and free server option cover exactly those two gaps. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check the current limits and data-handling terms in their docs before you point anything sensitive at it — free tiers change, and your cassette may contain real prompts.

Where free capacity is the right tool, and where it isn't

Situation	Free capacity	Why
Pinning cost logic in CI	Good fit	Replayed cassettes need tiny, predictable throughput
Cold-path dev loops and demos	Good fit	Low stakes, low volume, easy to re-run
A contractual p95 latency SLO	Wrong bet	You don't control a shared queue, so you can't promise a tail
Regulated or customer-identifiable data	Wrong bet	A free tier is the wrong place to discover a residency clause
One nightly batch dominating spend	Wrong bet	Per-success is the wrong metric; you want a hard per-job cap

The last row is the one people get wrong. Cost per success is a ratio, and ratios hide fixed cost. An idle server you keep warm all month contributes almost nothing to any single run and everything to the invoice.

Limitations worth stating out loud

A replay gate tests your cost logic, not your vendor's pricing. If the price per token moves, your cassette doesn't know, and your threshold silently drifts out of date — put an expiry date on the budget file. The divisor also punishes a flaky test corpus: make your fixture suite more failure-prone and cost per success rises even though production is fine. Keep the cassette small and curated, twenty to fifty runs, and review it like a fixture, not a log.

Don't adopt this if your real constraint is a latency SLO, if your data can't leave a controlled environment, or if your spend is dominated by a single large batch job rather than many small calls. And don't adopt it if nobody will own the thresholds — a red build with no owner trains the team to ignore red builds, which is worse than no gate at all.

Start with one command in report mode, one budget file, and one person who reads the failure message. If you want to run that command somewhere other than your metered build fleet, the free server option on MonkeyCode is a reasonable place to try it.