Your tests can make 66,000 paid LLM calls a month before your first user

You wrote tests for your LLM feature, and they call the real model, because that feels like the honest way to test it. Say there are 100 of them, one call each, and the suite runs 30 times a working day: locally while you work, and again on every push. That's 66,000 paid calls a month. On GPT-5.6 at list price it comes to $290, before a single user has signed up.

Every test run is a batch of paid calls

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send and output tokens are what the model writes back; output costs more. Prices are quoted per million tokens, and the ones vendors publish themselves are called list prices.

A test that calls the model is billed exactly like a user's request. The API can't tell them apart, and the 500th run of the same question is a new request. A vendor's prompt cache, a discount on input it has seen recently, can make the question cheaper, but the answer is written, and billed, again. So the monthly cost of your tests is calls per run × runs per day × working days × the price of one call. CI, the server that runs your tests on every push, grows the middle number without asking. A test watcher that reruns the suite on every file save grows it again.

My numbers are assumptions, not measurements: 100 model calls per run, 1,000 input and 200 output tokens each, 30 runs a working day, 22 working days a month. Put in your own.

Five models, one test suite

List prices come from the catalog of the LLM gateway I work on, read on 25 October 2026. The last column is what's left if the model is called live only once a night, which is where the next section ends up.

Model	List price in / out, $ per 1M	One test call	Every run, 66,000 calls a month	Live once a night, 3,000 calls
claude-haiku-5.5	0.10 / 0.50	$0.0002	$13.20	$0.60
gpt-5.6-mini	0.20 / 1.20	$0.00044	$29.04	$1.32
gemini-3.8-flash	0.75 / 3.75	$0.0015	$99.00	$4.50
gpt-5.6	2 / 12	$0.0044	$290.40	$13.20
claude-opus-5.5	4 / 20	$0.008	$528.00	$24.00

On the two smallest models the test bill is noise. On the models people pick for quality it's a real line in the budget, and it grows with every push instead of every user. Running the tests on a cheaper model than production lowers the price, but then you're testing a model you don't ship. The fix that keeps your production model is to stop paying for answers you already have.

Record each answer once, replay it after that

The idea is called record and replay: the first time a test sends a request, save the model's answer to a file, and every later run with the same request reads the file instead of calling the API. Libraries like vcrpy do this for HTTP requests in general. For one function, eleven lines of standard-library Python are enough:

import hashlib, json, pathlib

def recorded(call, folder="llm_cassettes"):
    def wrapper(**request):
        key = hashlib.sha256(json.dumps(request, sort_keys=True).encode()).hexdigest()
        path = pathlib.Path(folder) / f"{key[:16]}.json"
        if path.exists():                      # replay: no API call, no bill
            return json.loads(path.read_text())
        reply = call(**request)                # record: one paid call
        path.parent.mkdir(exist_ok=True)
        path.write_text(json.dumps(reply))
        return reply
    return wrapper

paid = []
ask = recorded(lambda **r: paid.append(r) or {"label": "positive"})
for _ in range(3):                             # three test runs
    ask(model="gpt-5.6", messages=[{"role": "user", "content": "Great phone!"}])
print(f"3 test runs, paid API calls: {len(paid)}")

The lambda stands in for the real API call, so the snippet runs as is and prints 3 test runs, paid API calls: 1. Run it again and it prints 0: the answer is already on disk. With the OpenAI SDK, the function you wrap is lambda **r: client.chat.completions.create(**r).model_dump().

Commit the folder, so CI replays the same answers. Change a prompt and its request no longer matches the saved one, so only that test pays again. Once a night, point folder at an empty directory and let every call reach the API: that's the 3,000-call column, under 5% of the calls. As a side effect, replayed runs can't fail because the API is slow, down or refusing requests at a busy hour.

What this doesn't solve

Replay tests your code, not the model. If the vendor changes how a model behaves, or you switch models, the saved answers won't notice until the nightly live run. A saved answer is also one sample: the model may word it differently next time, so live tests should check structure, like valid JSON with the right keys, rather than exact text. And the files hold whatever your test inputs held, so keep real customer data out of them.

A lower price per token is no substitute. The gateway I mentioned sells gpt-5.6 at $1.70 / $10.20 against the list $2 / $12, which takes the every-run bill from $290.40 to $246.84. Replay takes it to $13.20, at list price.

The takeaway

Your tests are your first heavy user. Multiply calls per run by runs per month before you wire a live model into CI, then record each answer once and pay for live calls only when you mean to.