DEV Community: Andrey Altrouter

Your LLM call works 99% of the time. Your 100-call job fails 63% of the time.

Andrey Altrouter — Tue, 03 Nov 2026 12:00:02 +0000

You write a script that summarizes a 100-page PDF, one LLM call per page. It dies on page 71 with a timeout. You run it again, and it starts from page 1. No single part is broken: the API had one bad second, which happens. But the rerun pays for pages 1 to 70 a second time, and its odds of dying are the same as the first run's.

One call is a coin toss, a job is a streak

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send, output tokens are what the model writes back. A call can fail with nobody at fault: a timeout, an overloaded server, an answer your code can't parse. Call the share of calls that succeed p. At p = 99%, a single call is safe enough.

A job finishes only if every call in it succeeds, so the odds multiply: a job of N calls finishes with probability p to the power N. Two calls at 99%: 0.99 × 0.99 ≈ 98%. A hundred calls: 0.99¹⁰⁰ = 36.6%, so the job fails in 63% of runs. At 95% per call, a hundred-call job finishes 0.6% of the time. Both rates are my assumptions, not measurements: 99% is a call that fails only on network trouble, 95% is closer to one whose answer must also parse as JSON. Short jobs barely notice: five calls at 99% still finish 95% of the time.

Starting over pays for finished pages again

What a failure costs depends on what your code does next. Start again from call 1, and every call before the failure is bought again, as many times as it takes. Retry only the call that failed, and nothing finished is paid for twice. Here are both, for jobs of 20 and 100 calls:

PRICE_IN, PRICE_OUT = 2.00, 12.00                   # GPT-5.6 list price, $ per 1M tokens
CALL = (1_000 * PRICE_IN + 250 * PRICE_OUT) / 1e6   # one page: $0.005

def job(n, p):
    finish = p ** n                                 # every call has to succeed
    start_over = (1 - finish) / ((1 - p) * finish)  # calls paid, restarting from call 1
    retry = n / p                                   # calls paid, retrying the failed call
    return finish, start_over, retry

for n, p in [(20, 0.99), (20, 0.95), (100, 0.99), (100, 0.95)]:
    finish, start_over, retry = job(n, p)
    print(f"{n:3} calls at {p:.0%}: finishes {finish:5.1%}, "
          f"start over ${start_over * CALL:5.2f}, retry ${retry * CALL:.2f}")

A page is 1,000 tokens in and 250 out. On GPT-5.6 at its list price, the one the vendor publishes, that's $2 per million input tokens and $12 per million output: $0.005 a page, $0.50 for the PDF. The prices come from the catalog of the LLM gateway I work on, read on 29 October 2026. The start_over line is the textbook formula for the expected wait until N successes in a row.

Job	Finishes in one run	Start over from call 1	Retry only the failed call
20 calls, 99% each	81.8%	$0.11, ×1.11	$0.10, ×1.01
20 calls, 95% each	35.8%	$0.18, ×1.79	$0.11, ×1.05
100 calls, 99% each	36.6%	$0.87, ×1.73	$0.51, ×1.01
100 calls, 95% each	0.6%	$16.79, ×33.6	$0.53, ×1.05

Each × compares with a run without errors: $0.10 for 20 pages, $0.50 for 100. Retrying costs the same small extra at any length, because you pay for N ÷ p calls. Starting over grows with the job. At 95%, the 100-page PDF takes 169 runs on average, and each failed run dies about 19 pages in: 3,358 paid calls, $16.79 for a $0.50 job. By hand you'd quit after a few runs, with nothing saved.

Retry the page, keep the pages

The fix has two parts. Retry the call that failed, a couple of times with a growing pause, instead of failing the whole job. And write each answer to disk as soon as you have it. Saved progress like this is a checkpoint: if the job dies anyway, the next run reads the file, skips finished pages and starts from the one that failed.

import json, pathlib, time

def run_job(pages, call_model, out=pathlib.Path("results.jsonl"), attempts=3):
    done = set()
    if out.exists():                      # an earlier run already paid for these
        done = {json.loads(line)["page"] for line in out.read_text().splitlines()}
    with out.open("a") as f:
        for i, page in enumerate(pages):
            if i in done:
                continue
            for attempt in range(attempts):
                try:
                    answer = call_model(page)
                    break
                except Exception:         # narrow this to timeouts and server errors
                    if attempt == attempts - 1:
                        raise             # give up; the next run starts from this page
                    time.sleep(2 ** attempt)
            f.write(json.dumps({"page": i, "answer": answer}) + "\n")
            f.flush()                     # on disk before the next paid call

call_model is your own function: it sends one page to the model and returns the answer as text. Each line of results.jsonl is one finished page, so a crash costs at most the call in progress, and the next run carries on from there. The job now costs N ÷ p calls at any length: 105 for the PDF at 95%.

Why not catch the error and skip the page? The job would always finish and never pay twice, but at 95% about 5 pages in 100 would be quietly missing, and the summary would have holes you can't see.

What this doesn't solve

A page too long for the model fails on every attempt: let the job stop, fix the cause, and the checkpoint keeps the rest. The math treats failures as independent, but in an outage they come together; that's why the snippet waits longer each time and then gives up. Retry only idempotent steps, ones that are safe to run twice: a call that also emails a customer isn't. And no gateway can do this for you, including the one I work on: it sees single calls, not your job, so it can't know which 70 pages you already have.

The takeaway

A job of N calls finishes with probability p to the power N. Make one call, not the whole job, the thing you retry, and save each answer before you pay for the next.

Your LLM cost log says $0.00. The invoice says $1,320.

Andrey Altrouter — Mon, 02 Nov 2026 12:00:02 +0000

You log what each LLM call costs, right next to the request, in a DECIMAL(10,2) column like every other amount of money in your app. A month later the dashboard says the new feature cost $0.00. The vendor's invoice says $1,320. Nothing leaked. One call cost less than half a cent, and your database rounded it away.

A cent is too big for one LLM call

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send and output tokens are what the model writes back; output costs more. Prices are quoted per million tokens, and the ones vendors publish themselves are called list prices. Every response also carries a usage field: the exact number of input and output tokens you were billed for.

Take a modest call: 1,000 tokens in, 200 out. On GPT-5.6, at $2 per million input tokens and $12 per million output, that's $0.002 + $0.0024 = $0.0044. A money column keeps two digits after the point, so anything under half a cent becomes $0.00. PostgreSQL does exactly that when you insert 0.0044 into a DECIMAL(10,2) column, and raises no error. The habit comes from payments, where an amount is a whole number of cents. A single LLM call is usually a fraction of one.

Four of five models log as free

I priced the same call on five models. List prices come from the catalog of the LLM gateway I work on, read on 27 October 2026. A month here is 10,000 calls a day for 30 days.

Model, list price per 1M in / out	One call costs	Logged as	Month, spent	Month, logged
claude-haiku-5.5, $0.10 / $0.50	$0.0002	$0.00	$60	$0
gpt-5.6-mini, $0.20 / $1.20	$0.00044	$0.00	$132	$0
gemini-3.8-flash, $0.75 / $3.75	$0.0015	$0.00	$450	$0
gpt-5.6, $2 / $12	$0.0044	$0.00	$1,320	$0
claude-opus-5.5, $4 / $20	$0.008	$0.01	$2,400	$3,000

Four models log as free. The fifth, Claude Opus 5.5, logs 25% too much, because $0.008 rounds up to a whole cent. Don't count on the errors cancelling out: one feature sends calls of roughly the same size, so they all round the same way. Your vendor's dashboard still shows the right total, but it can't tell you which feature or which user spent it. Only your log can, and here it says nobody did.

The log can also jump when almost nothing changed. On GPT-5.6 with 200 output tokens, a call with 1,200 input tokens costs $0.0048 and logs as $0.00; with 1,400 it costs $0.0052 and logs as $0.01. Make your instructions 200 tokens longer, and the monthly line goes from $0 to $3,000 while the real bill moves from $1,440 to $1,560.

Store millionths, round once

The fix is to round where you read, not where you write. Two ways work. Store the raw facts, the model name and the two token counts from usage, and turn them into money when you build the report; when a price changes, you can redo the math. Or store the cost as a whole number of micro-dollars, millionths of a dollar. Tokens times the price per million already gives micro-dollars: 1,000 × $0.20 + 200 × $1.20 is 440 micro-dollars for one GPT-5.6 mini call.

PRICES = {  # list price, USD per 1M tokens: (input, output), 27 Oct 2026
    "claude-haiku-5.5": (0.10, 0.50),
    "gpt-5.6-mini": (0.20, 1.20),
    "gemini-3.8-flash": (0.75, 3.75),
    "gpt-5.6": (2.00, 12.00),
    "claude-opus-5.5": (4.00, 20.00),
}

def cost_micro_usd(model: str, input_tokens: int, output_tokens: int) -> int:
    """Cost of one call in millionths of a dollar. Store this, not cents."""
    price_in, price_out = PRICES[model]
    return round(input_tokens * price_in + output_tokens * price_out)

CALLS_A_DAY = 10_000
for model in PRICES:
    micro = cost_micro_usd(model, 1_000, 200)
    logged = round(micro / 1_000_000, 2) * CALLS_A_DAY  # round each call, then add
    spent = micro * CALLS_A_DAY / 1_000_000             # add, then convert once
    print(f"{model:17} logged ${logged:7.2f}   spent ${spent:7.2f}   a day")

It prints $0.00 logged against $2.00, $4.40, $15.00 and $44.00 actually spent a day on the first four models, and $100 against $80 on Opus. In your app, feed it the two counts from each response's usage: prompt_tokens and completion_tokens in OpenAI-style APIs. In the database, that's a BIGINT (64-bit integer) column, say cost_micro_usd, which holds up to $9.2 trillion. A float column would keep the fractions too, but floats added in a different order can give slightly different totals; integers add up exactly.

What this doesn't solve

An exact sum of your rows still isn't the invoice. It's only as right as your token counts and your price table. Discounted tokens, such as a repeated prompt the vendor serves from its cache, have their own price; give them their own column. A call that times out mid-answer may return no usage, and then your log never learns what it cost. And no gateway can fix this for you, including the one I work on: the rounding happens in your database, after the response has arrived. Once a month, compare your sum with the vendor's invoice.

The takeaway

A typical LLM call costs less than the smallest amount your money column can hold. Log tokens or micro-dollars for every call, and round once, when you add them up.

Your LLM judge can cost 9.5x the chatbot it grades. Grade 400 replies a day, not all of them.

Andrey Altrouter — Sun, 01 Nov 2026 12:00:02 +0000

You shipped a support chatbot on GPT-5.6 mini, and now you want to know how often it answers wrong. The usual advice is to let a stronger model grade every reply. I priced that setup: each grade costs 9.5 times the reply it grades. At 10,000 replies a day, the bot costs $132 a month to run and $1,260 a month to watch. Most of those grades tell you little that 400 wouldn't.

Grading a reply costs more than writing it

An LLM judge is a second model that reads a reply and grades it against your instructions: did the bot answer the question, did it make up a refund policy. Three things make it expensive. It reads everything the bot read, plus the reply, plus the grading instructions. It is usually a stronger model, because a judge no smarter than the bot tends to miss the bot's mistakes. And the usual setup calls it on every reply.

My numbers are assumptions, not measurements. The bot reads 1,000 tokens and writes 200. The judge reads those 1,200 plus 300 tokens of instructions and writes a 100-token verdict. List prices come from the catalog of the LLM gateway I work on, read on 26 October 2026: $0.20 / $1.20 per million for GPT-5.6 mini, $2 / $12 for GPT-5.6. A reply costs $0.00044. Its grade costs $0.0042.

Precision depends on the count, not on the traffic

The user already has the reply, so grading it fixes nothing. What you're after is one number: the share of bad replies. A random sample, a few replies picked by chance, measures that number too. How close it lands is the margin of error: 95% of the time, what you measure is within that many points of the true rate. For a 5% failure rate it equals 1.96 × √(0.05 × 0.95 / n), where n is the number of graded replies. Your traffic isn't in the formula.

Grades a day	Margin on a 5% failure rate	Judge on GPT-5.6, $ a month	Judge vs the bot's $132
200	±3.0 points	$25.20	0.19x
400	±2.1 points	$50.40	0.38x
1,600	±1.1 points	$201.60	1.5x
10,000 (every reply)	±0.4 points	$1,260.00	9.5x

Four times the grades buy half the margin. Grading every reply instead of 400 costs 25 times more and narrows "somewhere between 2.9% and 7.1%" to "between 4.6% and 5.4%". At a million replies a day, the same 400 grades still give ±2.1 points, and the judge shrinks to 0.4% of the bot's bill.

Grade a fixed number a day

So pick the precision first, turn it into a daily count, and let the chance of grading follow your traffic:

import math, random

GRADES_PER_DAY = 400

def should_grade(replies_yesterday: int) -> bool:
    """Pick about GRADES_PER_DAY random replies a day, whatever the traffic."""
    return random.random() < min(1.0, GRADES_PER_DAY / max(replies_yesterday, 1))

def margin(failure_rate: float, graded: int) -> float:
    """How far the measured rate can be from the true one, 95% of the time."""
    return 1.96 * math.sqrt(failure_rate * (1 - failure_rate) / graded)

for n in (200, 400, 1600, 10_000):
    print(f"{n:>6} grades a day: 5% ± {margin(0.05, n):.1%}")

It prints the margin column of the table. Call should_grade after each reply with yesterday's reply count, and send the chosen replies to the judge in the background, so no user waits for it. Don't grade the first 400 replies of the day instead: people who write at night ask different questions. If traffic doubles before the count catches up, you grade 800 and pay $3.36 for the day instead of $1.68. Need tighter numbers? Pool a week: 2,800 grades give ±0.8 points at the same daily cost. On a small bot with 2,000 replies a day, 400 grades still cost 1.9 times the bot itself; settle for 200 and ±3 points if that's too much.

What this doesn't solve

A sample measures the rate; it doesn't stop a bad reply. If one must never reach a user, you need a check before every reply goes out, paid on every reply. Rare failures slip through too: at a true rate of 0.1%, two days out of three a 400-reply sample contains none, so catch things like a leaked email address with a cheap code check over all traffic. The judge is a model as well; compare it with 50 replies you graded by hand before trusting its number. And no gateway does this for you: the one I work on logs each call's status code, and a 200 means the model answered, not that it answered right.

The takeaway

Decide how precisely you need to know your failure rate, turn that into a daily number of grades, and grade that many random replies. Your traffic will grow; the judge bill doesn't have to.

Your tests can make 66,000 paid LLM calls a month before your first user

Andrey Altrouter — Sat, 31 Oct 2026 12:00:02 +0000

You wrote tests for your LLM feature, and they call the real model, because that feels like the honest way to test it. Say there are 100 of them, one call each, and the suite runs 30 times a working day: locally while you work, and again on every push. That's 66,000 paid calls a month. On GPT-5.6 at list price it comes to $290, before a single user has signed up.

Every test run is a batch of paid calls

A test that calls the model is billed exactly like a user's request. The API can't tell them apart, and the 500th run of the same question is a new request. A vendor's prompt cache, a discount on input it has seen recently, can make the question cheaper, but the answer is written, and billed, again. So the monthly cost of your tests is calls per run × runs per day × working days × the price of one call. CI, the server that runs your tests on every push, grows the middle number without asking. A test watcher that reruns the suite on every file save grows it again.

My numbers are assumptions, not measurements: 100 model calls per run, 1,000 input and 200 output tokens each, 30 runs a working day, 22 working days a month. Put in your own.

Five models, one test suite

List prices come from the catalog of the LLM gateway I work on, read on 25 October 2026. The last column is what's left if the model is called live only once a night, which is where the next section ends up.

Model	List price in / out, $ per 1M	One test call	Every run, 66,000 calls a month	Live once a night, 3,000 calls
claude-haiku-5.5	0.10 / 0.50	$0.0002	$13.20	$0.60
gpt-5.6-mini	0.20 / 1.20	$0.00044	$29.04	$1.32
gemini-3.8-flash	0.75 / 3.75	$0.0015	$99.00	$4.50
gpt-5.6	2 / 12	$0.0044	$290.40	$13.20
claude-opus-5.5	4 / 20	$0.008	$528.00	$24.00

On the two smallest models the test bill is noise. On the models people pick for quality it's a real line in the budget, and it grows with every push instead of every user. Running the tests on a cheaper model than production lowers the price, but then you're testing a model you don't ship. The fix that keeps your production model is to stop paying for answers you already have.

Record each answer once, replay it after that

The idea is called record and replay: the first time a test sends a request, save the model's answer to a file, and every later run with the same request reads the file instead of calling the API. Libraries like vcrpy do this for HTTP requests in general. For one function, eleven lines of standard-library Python are enough:

import hashlib, json, pathlib

def recorded(call, folder="llm_cassettes"):
    def wrapper(**request):
        key = hashlib.sha256(json.dumps(request, sort_keys=True).encode()).hexdigest()
        path = pathlib.Path(folder) / f"{key[:16]}.json"
        if path.exists():                      # replay: no API call, no bill
            return json.loads(path.read_text())
        reply = call(**request)                # record: one paid call
        path.parent.mkdir(exist_ok=True)
        path.write_text(json.dumps(reply))
        return reply
    return wrapper

paid = []
ask = recorded(lambda **r: paid.append(r) or {"label": "positive"})
for _ in range(3):                             # three test runs
    ask(model="gpt-5.6", messages=[{"role": "user", "content": "Great phone!"}])
print(f"3 test runs, paid API calls: {len(paid)}")

The lambda stands in for the real API call, so the snippet runs as is and prints 3 test runs, paid API calls: 1. Run it again and it prints 0: the answer is already on disk. With the OpenAI SDK, the function you wrap is lambda **r: client.chat.completions.create(**r).model_dump().

Commit the folder, so CI replays the same answers. Change a prompt and its request no longer matches the saved one, so only that test pays again. Once a night, point folder at an empty directory and let every call reach the API: that's the 3,000-call column, under 5% of the calls. As a side effect, replayed runs can't fail because the API is slow, down or refusing requests at a busy hour.

What this doesn't solve

Replay tests your code, not the model. If the vendor changes how a model behaves, or you switch models, the saved answers won't notice until the nightly live run. A saved answer is also one sample: the model may word it differently next time, so live tests should check structure, like valid JSON with the right keys, rather than exact text. And the files hold whatever your test inputs held, so keep real customer data out of them.

A lower price per token is no substitute. The gateway I mentioned sells gpt-5.6 at $1.70 / $10.20 against the list $2 / $12, which takes the every-run bill from $290.40 to $246.84. Replay takes it to $13.20, at list price.

The takeaway

Your tests are your first heavy user. Multiply calls per run by runs per month before you wire a live model into CI, then record each answer once and pay for live calls only when you mean to.

A $5-a-month AI plan is a 17-cent daily budget. On GPT-5.6 that's 25 messages.

Andrey Altrouter — Fri, 30 Oct 2026 12:00:02 +0000

You've built a chatbot and plan to charge $5 a month for it. One message on GPT-5.6 costs well under a cent, so unlimited use sounds safe. Divide the $5 by 30 days, though, and each subscriber gets about 17 cents of model calls a day. A typical message costs 0.66 cents at OpenAI's published price, so from the 26th message of the day, that user costs you more than they pay.

A monthly price is a daily budget

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send: your instructions, the recent chat history and the user's question. Output tokens are what the model writes back, and they cost more. Prices are quoted per million tokens, and the ones vendors publish themselves are called list prices.

So one message costs input tokens × input price + output tokens × output price. For this article I fixed a message at 1,500 input tokens and 300 output: a few lines of instructions, a few earlier turns and an answer of two paragraphs. On GPT-5.6, at $2 per million input tokens and $12 per million output, that's $0.003 + $0.0036 = $0.0066.

A flat monthly price is a daily budget per user: $5 / 30 = $0.167. Divide it by the cost of one message and you get how many messages a day the plan pays for. That number still leaves nothing for hosting, card fees or profit.

Five models, one $5 plan

I ran that division for five models from three vendors. List prices come from the catalog of the LLM gateway I work on, read on 24 October 2026. The last column assumes you keep 70% of the $5 as margin, the share of revenue left after paying for the model, so the model gets $1.50 a month, 5 cents a day.

Model	List price in / out, $ per 1M	One message	Messages a day for $5	Keeping 70%
claude-haiku-5.5	0.10 / 0.50	$0.0003	555	166
gpt-5.6-mini	0.20 / 1.20	$0.00066	252	75
gemini-3.8-flash	0.75 / 3.75	$0.00225	74	22
gpt-5.6	2 / 12	$0.0066	25	7
claude-opus-5.5	4 / 20	$0.012	13	4

A message costs 40 times more in the last row than in the first, and the same $5 buys 40 times fewer of them. With a margin, a GPT-5.6 user who sends eight messages a day is already over budget.

Most of your users won't send 25 messages a day, and a flat price is set for the average user. The budget is for the one who isn't average. Someone who wires your bot into a script and sends 1,000 messages a day costs $6.60 a day on GPT-5.6, $198 a month, and pays you $5.

Count dollars per user, not messages

A message limit is easy to write and wrong in both directions: "hi" and a pasted 20-page document count the same. Count dollars instead. The API reports what each call billed in a usage field: prompt_tokens and completion_tokens. Price them, add them to the user's total for the day, and refuse new messages once the total reaches the budget.

from collections import defaultdict
from datetime import date

PLAN_USD, KEEP = 5.00, 0.70         # monthly price, share you keep as margin
PRICE_IN, PRICE_OUT = 2.00, 12.00   # $ per 1M tokens, gpt-5.6 list price
DAILY_USD = PLAN_USD * (1 - KEEP) / 30
spent = defaultdict(float)          # (user, day) -> dollars spent on the model

def allowed(user):
    return spent[user, date.today()] < DAILY_USD

def record(user, prompt_tokens, completion_tokens):
    cost = (prompt_tokens * PRICE_IN + completion_tokens * PRICE_OUT) / 1e6
    spent[user, date.today()] += cost

n = 0
while allowed("alice"):             # one simulated day of 1,500-in / 300-out messages
    record("alice", 1500, 300)
    n += 1
print(f"${DAILY_USD:.4f} a day, {n} messages, spent ${spent['alice', date.today()]:.4f}")

It needs no API key: the loop at the bottom stands in for a day of GPT-5.6 messages and prints $0.0500 a day, 8 messages, spent $0.0528. In your app, call allowed(user) before each request and record(user, r.usage.prompt_tokens, r.usage.completion_tokens) after it, where r is the API response. The eighth message gets through because the check runs before the reply is priced, so a user can go over by one reply at most.

Keep spent in your database rather than in memory, or every restart gives everyone a fresh day. When a user hits the limit, say so and offer something: a reset at midnight, a cheaper model for the rest of the day, or a higher plan.

What this doesn't solve

1,500 tokens in and 300 out is my assumption, not a measurement: in a long chat the input grows every turn, so take averages from usage on real traffic. The check runs before a call and the charge after, so one 100,000-token paste costs $0.20 on GPT-5.6, four days of the 5-cent budget, in a single message. Cap the size of a single request too. And the table says nothing about quality: a model 40 times cheaper may not do your job.

A lower price per token barely moves any of this. The gateway I mentioned sells GPT-5.6 15% under list, at $1.70 / $10.20, which lifts the $5 plan from 25 to 29 messages a day. Switching to gpt-5.6-mini lifts it to 252.

The takeaway

A flat price doesn't make usage free. It gives every user a daily budget: divide the price by 30, divide again by what a message costs, and enforce the result in dollars, not messages.

Python wrote 19.944444444444446 and the model billed 7 tokens for it

Andrey Altrouter — Thu, 29 Oct 2026 12:00:02 +0000

You want a model to look over a day of sensor readings and flag anything odd. One reading a minute is 1,440 rows, and you write them out with Python's csv module, the obvious way. That file is 49,723 tokens. The same information fits in 17,287. The other 32,436 tokens are digits that either repeat on every row or mean nothing.

The model reads digits three at a time

You pay per token, a short chunk of text the model reads as one piece. Input tokens are what you send, output tokens are what it writes back, and each has its own price per million. A tokenizer is the program that does the cutting. I used tiktoken, OpenAI's open-source one, with its o200k_base encoding.

For numbers the rule is mechanical: digits are cut into groups of up to three, left to right. 19.94 is three tokens: 19, ., 94. 19.944444444444446 is seven: 19, ., 944, 444, 444, 444, 446.

That second number is what Python gives you for (67.9 - 32) * 5 / 9, a Fahrenheit reading converted to Celsius. A floating-point number is stored in binary, most decimal fractions can't be stored exactly, and Python prints the shortest form that reads back to the same value, often 17 digits. str(), f-strings, csv.writer and json.dumps all print that full form. The sensor measured to 0.1 °F, so four digits carry the measurement and thirteen are leftovers of arithmetic.

The timestamps cost more than the readings

I built one synthetic day: a greenhouse sensor, one row per minute, three columns. Temperature is the converted reading above. Humidity is the average of three whole-number samples, which is where 50.666666666666664 comes from. Timestamps are isoformat() strings with microseconds, as datetime.now() produces them. Then I counted the file four ways.

What you send, 1,440 rows	Tokens	Per row	Saved
`csv.writer` default	49,723	34.5	—
Values rounded to 2 decimals	40,327	28.0	19%
Timestamps as `HH:MM`, values as is	26,683	18.5	46%
Both	17,287	12.0	65%

I expected the long floats to be the problem. They're the smaller half. The timestamp column alone is 28,799 tokens, 58% of the file. 2026-10-22T00:00:00.670487+00:00 is 19 tokens, and 16 of them are a date and timezone that repeat on every row plus seconds and microseconds that only record when the logger woke up. 00:00 is 3 tokens. That long form is ISO 8601, the standard date-time format, and it's the right choice for storage. In a prompt where every row shares one date, it's a tax.

Rounding to one decimal buys nothing

With short timestamps in place, I tried rounding harder. One, two and three decimals all came out at exactly 17,287 tokens. That's the three-digit rule again: 19.9, 19.94 and 19.944 are each 19, . and one more token. I checked all 1,110 digit strings of length one to three, and every one is a single token. The fourth decimal is where the next token starts.

So one decimal only costs you data. Two decimals kept all 96 distinct temperatures in the file. One decimal merged them into 53, because 0.1 °C is coarser than the sensor's 0.1 °F. The rule: round to the precision your source has, and don't expect tokens back below three decimals. To check, count distinct values before and after rounding. If the count drops, you rounded away a measurement.

Shorten the rows before the call

Save your readings the way your code writes them and run python compact_csv.py readings.csv. It expects an ISO timestamp in the first column and numbers in the rest:

import csv, sys, tiktoken  # pip install tiktoken
from datetime import datetime

DECIMALS = 2  # the precision your data really has; 1-3 cost the same tokens
src = open(sys.argv[1], encoding="utf-8").read()
header, *rows = csv.reader(src.splitlines())
first = datetime.fromisoformat(rows[0][0])
lines = [f"# {first:%Y-%m-%d} {first:%Z}, time is HH:MM", ",".join(header)]
for ts, *values in rows:
    hhmm = f"{datetime.fromisoformat(ts):%H:%M}"
    lines.append(",".join([hhmm] + [str(round(float(v), DECIMALS)) for v in values]))
compact = "\n".join(lines)

tok = tiktoken.get_encoding("o200k_base").encode
print(f"{len(tok(src)):,} tokens as written, {len(tok(compact)):,} compact")
print(lines[0], lines[2], sep="\n")

It writes the date and timezone once in a header line, keeps hours and minutes, rounds the values and prints both counts. On my test file it prints 49,723 tokens as written, 17,302 compact; the header line is the 15 tokens over the table. The compact variable is what you send.

Now the bill. Say ten sensors, each checked every hour against its last 24 hours: 240 calls a day, each with a 41-token instruction and an answer of about 200 tokens. Prices are gpt-5.6's list prices, $2 per million input tokens and $12 per million output, from the catalog of the LLM gateway I work on, read on 23 October 2026. As written, a call costs $0.10 and the month $734. Compact, it's $0.037 a call and $267 a month. Input is 98% of the first bill, so the answer length barely matters here. On gpt-5.6-mini, ten times cheaper, the month is $73 against $27.

What this doesn't solve

Rounding destroys information when the digits are real: money (keep the cents, and keep money out of floats altogether), GPS coordinates, where the fifth decimal is about a metre, or lab measurements. HH:MM only works while the window sits inside one day and one timezone. A window across midnight needs the date where it changes, or minutes since the window start. The data is synthetic, one simulated sensor, and the counts come from tiktoken, not from an API's usage field. Claude and Gemini use their own tokenizers, so count yours with your vendor's token counter. Sending min, max and mean computed in code is cheaper still, though then the model no longer sees each reading.

The gateway sells gpt-5.6 at $1.70 / $10.20, 15% under list, which makes the two monthly bills $624 and $227. It bills the tokens you send, digits included, and rounds nothing for you.

The takeaway

The tokenizer can't tell a measurement from arithmetic noise, and it bills both. Write the date once and round to the precision you have, and a day of readings costs a third of what Python's defaults send.

Two-thirds of a web page's tokens are markup, and most of it isn't the menu

Andrey Altrouter — Wed, 28 Oct 2026 12:00:01 +0000

You build a small "ask the docs" helper: download a page, paste it into the prompt, add the user's question. It works, and the bill comes out about three times what the text on the screen suggests. The Node.js reference page for UDP sockets is 7,127 tokens of text. Downloaded as is, it's 23,178.

The model reads the tags too

You pay per token, a chunk of text about three-quarters of an English word long. Input tokens are what you send, output tokens are what the model writes back. What your code downloads isn't the page you see, it's the HTML behind it: tags, attributes, inline scripts, SVG icons. The model has no "skip the markup" mode. Every character you send is cut into tokens and billed at the input price.

Markup is made of short, rare fragments, so each token covers fewer characters than in prose: 3.4 against 4.1 on the pages below. Here is one heading from the Node.js path page, exactly as the server sends it:

<section><h3>Windows vs. POSIX<span><a class="mark" href="#windows-vs-posix" id="windows-vs-posix">#</a></span><a aria-hidden="true" class="legacy" id="path_windows_vs_posix"></a></h3>

That's 62 tokens. The words a reader sees, "Windows vs. POSIX", are 5. Code samples pay as well: every apostrophe in them is written as ', an HTML entity (the escaped form of a character), and each one adds about 2.5 tokens over a plain '.

I counted 63 pages four ways

I took all 63 API reference pages of Node.js 20.19.2, the HTML copies that ship with its documentation on Debian, and counted each one four ways with tiktoken, OpenAI's open-source tokenizer (the program that cuts text into tokens), using o200k_base, the encoding of current GPT models.

What you send, all 63 pages	Tokens	× text	Saved
Raw HTML, as downloaded	2,284,043	3.06×	—
Only the article block, tags kept	1,754,618	2.35×	23%
Whole page, tags stripped	746,178	1.00×	67%
Article block, tags stripped	705,226	0.95×	69%

The obvious fix is to cut away the page chrome, the navigation, header and footer repeated on every page. Here it saves 23%. Stripping the tags saves 67%, because most of the waste sits inside the article. Against the bare article text, the raw pages carry 1.58 million extra tokens: 34% around the article and 66% inside it. The 63 pages hold 34,912 href= attributes, 33,092 class= attributes and 22,622 escaped apostrophes.

Per page, raw HTML ran from 2.05× its text (C++ addons) to 9.2× (the short usage example), plus one stub page at 20.7×. 50 of the 63 landed between 2.5× and 4.5×.

What it does to the bill

Take the UDP page, close to the median at 3.25×, and a helper that answers 1,000 questions a day: one page, a 30-token question and a 300-token answer per call. Prices are gpt-5.6's list prices, $2 per million input tokens and $12 per million output, from the catalog of the LLM gateway I work on, read on 22 October 2026.

With raw HTML the month costs $1,500. With text it costs $537. The answer is the same $108 either way, 7% of the first bill and 20% of the second. Cutting away only the navigation and keeping the tags lands at $1,070.

So the lever here is input. It's 93% of the raw bill, and more than two-thirds of that input is markup the model didn't need to answer a question about UDP sockets.

Strip the tags before the call

Download the page the way your code will, curl -o page.html <url>, and run python page_text.py page.html. It needs the standard library plus tiktoken for the count:

import sys, tiktoken, html.parser  # pip install tiktoken
SKIP = {"head", "script", "style", "svg", "nav", "header", "footer", "aside"}
BLOCK = {"p", "div", "li", "pre", "tr", "td", "th", "br", "h1", "h2", "h3", "h4", "section"}
class PageText(html.parser.HTMLParser):
    def __init__(self):
        super().__init__()
        self.skip, self.parts = 0, []  # skip > 0 means "inside a dropped tag"
    def handle_starttag(self, tag, attrs):
        self.skip += tag in SKIP
        self.parts.append("\n" if tag in BLOCK else "")
    def handle_endtag(self, tag):
        self.skip -= tag in SKIP
    def handle_data(self, data):
        if not self.skip: self.parts.append(data)

page = open(sys.argv[1], encoding="utf-8").read()
p = PageText(); p.feed(page)
text = "\n".join(s.strip() for s in "".join(p.parts).splitlines() if s.strip())
tok = tiktoken.get_encoding("o200k_base").encode
print(f"{len(tok(page)):,} tokens as HTML, {len(tok(text)):,} as text")

It drops what rarely holds the answer (scripts, styles, icons, navigation, header, footer), keeps the rest as text, and starts a new line at every paragraph, list item, heading and table cell, so code samples stay readable. On the UDP page it prints 23,178 tokens as HTML, 7,127 as text. The text variable is what you send.

The Node.js sidebar isn't a <nav> tag, so its list of modules stays in. If a site wraps the article in <main> or gives it an id, cut that block out first; here that removed the sidebar and the table of contents, about 5% of the text tokens.

What this doesn't solve

Stripping throws away what tags carry. Links lose their URLs and tables lose their columns, so if the task is "list every link" or "read this price table", keep that part as HTML or convert it to Markdown. Pages built by JavaScript in the browser can arrive as a nearly empty shell, and no stripping helps there. Documentation pages are lean by web standards, I measured local copies rather than the live site, and I didn't measure any other kind of site, so run the script on yours. The counts come from OpenAI's tokenizer; Claude and Gemini split text differently.

The gateway I mentioned sells gpt-5.6 at $1.70 / $10.20, 15% under list, which turns the two monthly bills into $1,275 and $457. That's the same 15% off both. It bills the tokens you send, tags included, and strips nothing for you.

The takeaway

The model bills markup at the same price as meaning. Send it the text of the page, not the page: on these 63 pages that was two-thirds of the input, for twenty lines of code.

Ten reviews per LLM call cut the bill by 57%. A thousand per call only reaches 65%.

Andrey Altrouter — Tue, 27 Oct 2026 12:00:01 +0000

You have 100,000 customer reviews and a prompt that tags one of them as positive or negative and says what it's about. Sending them one by one is the obvious first version. Sending ten per request cuts the bill by 57%. Sending a thousand per request cuts it by 65%, and those last eight points are where the trouble starts.

You pay for the rules once per request

You pay per token, a chunk of text about three-quarters of an English word. Input tokens are what you send, output tokens are what the model writes back, and output usually costs several times more.

A labeling request has two parts: the instructions (the labels, the rules, a couple of examples) and the review itself. The API remembers nothing between calls, so the instructions travel with every request. In my test, a one-review request sends 313 tokens, and 295 of them are the same every time: the rules plus a few tokens of message formatting. Label 100,000 reviews one at a time and you pay for those rules 100,000 times.

Pack ten reviews into one request and the rules are paid once per ten. Each review now costs a tenth of the instructions, its own text, a short [7] label in front of it, and its line of the answer. The first part shrinks as you pack more. The rest doesn't shrink at all.

Where the curve goes flat

My test task is a made-up clothing store with two labels per review: sentiment (positive, negative, mixed) and topic (shipping, quality, sizing, price, support, other). I wrote 20 short reviews and their correct answers myself, and counted everything with tiktoken, OpenAI's open-source token counter, using o200k_base, the encoding of current GPT models. I didn't call a model: the replies are the right answers in the requested format. Prices are gpt-5.6's list prices, the ones the vendor publishes, $2 per million input tokens and $12 per million output, from the catalog of the LLM gateway I work on, read on 21 October 2026.

Reviews per call	Tokens per review, in / out	Tokens in one reply	100,000 reviews	Saved
1	313 / 13	13	$78.38	—
5	90 / 19	96	$41.00	48%
10	54 / 19	192	$33.82	57%
100	24 / 19	1,920	$27.83	64%
1,000	21 / 19	19,201	$27.25	65%

Even with free instructions the bill can't drop below $27.18, because the reviews and the answers stay. So 65% is the ceiling, and ten per call already covers 87% of the way there. Packing also adds a cost: each answer line needs an "id", so output per review grows from 13 tokens to 19, at six times the input price. At one per call output was 20% of the bill. At ten it's 68%, and packing can't touch that part.

A bigger request means a bigger failure

With one review per call, a bad reply loses one review. With a hundred per call, a reply that stops halfway can lose all hundred, and you still pay for the half that arrived. Replies break for boring reasons: the connection drops, the model skips an item, or it hits max_tokens, the cap you set on how long its answer may be.

The answer format decides how much you keep. I cut a ten-review reply at 70% of its length. Written as one JSON array, it doesn't parse at all, so 0 of 10 labels survive. Written as JSON Lines, one JSON object per line, 6 of 10 lines parse and only 4 reviews go back for a second try. Both formats cost the same: 194 and 192 tokens.

So ask for one object per line that carries the id, set max_tokens to the number of items times one answer line with room to spare, and check finish_reason, the field in the response that reads length when the reply hit the cap.

Find your own flat spot

Count your instructions, one item and one answer line on your real prompt, then plug them in:

# Token counts from your own prompt, prices in $ per 1M tokens
INSTR_1, INSTR_N = 295, 325  # instructions: one-item version, packed version
ITEM, TAG = 18, 3            # tokens per item, plus its "[7] " label
OUT_1, OUT_N = 13, 19        # answer per item: alone, and with its "id" key
PRICE_IN, PRICE_OUT = 2.00, 12.00
ITEMS = 100_000

def bill(n):
    if n == 1:
        tokens_in, tokens_out = INSTR_1 + ITEM, OUT_1
    else:
        tokens_in, tokens_out = INSTR_N / n + ITEM + TAG, OUT_N
    return ITEMS * (tokens_in * PRICE_IN + tokens_out * PRICE_OUT) / 1e6

for n in (1, 5, 10, 20, 100):
    print(f"{n:>4} per call: ${bill(n):6.2f}  ({1 - bill(n) / bill(1):.0%} saved)")

It prints $78.20 at one per call, $33.50 at ten and $27.65 at a hundred, within 35 cents of the table because the inputs are rounded. Stop where doubling the batch adds only a couple of points; here that's ten to twenty. This isn't a vendor's batch API, which gives a discount for waiting hours for results but still sends one request per review, rules included. The two stack.

Then read the reply line by line and resend only what's missing:

import json

def parse_packed(reply, sent_ids):
    labels = {}
    for line in reply.splitlines():
        try:
            item = json.loads(line)
            labels[item["id"]] = (item["sentiment"], item["topic"])
        except (ValueError, KeyError, TypeError):
            continue  # cut-off or malformed line: that item gets resent
    missing = [i for i in sent_ids if i not in labels]
    return labels, missing

reply = '''{"id": 1, "sentiment": "negative", "topic": "sizing"}
{"id": 2, "sentiment": "negative", "topic": "shipping"}
{"id": 3, "sentiment": "posi'''
labels, missing = parse_packed(reply, [1, 2, 3, 4])
print(f"parsed {len(labels)}, resend {missing}")

It prints parsed 2, resend [3, 4].

What this doesn't solve

I didn't measure accuracy. Models can skip, merge or mislabel items in a long list, and my replies are perfect by construction, so compare a sample of packed results with one-at-a-time ones before you switch. The counts come from OpenAI's tokenizer, and Claude and Gemini split text differently. gpt-5.6 can also spend hidden thinking tokens, billed as output, that none of these numbers include. And short reviews favour packing: the longer each item, the smaller the instructions' share and the smaller the saving.

The gateway I mentioned sells gpt-5.6 at $1.70 / $10.20, 15% under list. That takes ten per call from $33.82 to $28.75 and leaves the curve where it was. It also fills in a max_tokens of 4,096 when a request has none, which the official API doesn't, so at 19 tokens per review a call with more than about 213 reviews gets cut off there.

The takeaway

Instructions are billed per request, items per item. Pack enough to turn the instructions into a rounding error, about ten here, and stop: past that point you're buying risk, not savings.

A cheap model that handles 70% of your requests can cut the bill by 65% — or raise it by 5%

Andrey Altrouter — Mon, 26 Oct 2026 12:00:02 +0000

A popular way to cut an LLM bill is to send every request to a small, cheap model first and hand it to the big one only when the small one gets it wrong. I priced that setup on five pairs of models from the same vendor, with the small model handling 7 requests out of 10. On one pair it cuts the bill by 65%. On another it makes the bill 5% bigger. Same success rate, opposite outcome.

The failed attempt is billed too

You pay per token, a chunk of text about three-quarters of an English word, at one price for input (what you send) and a higher one for output (what the model writes back). The per-token prices vendors publish are called list prices and are quoted per million tokens.

The setup above is called a cascade: try the cheap model, check its answer, and escalate to the strong model only if the check fails. The detail that's easy to miss is that the cheap attempt is billed whether it passes or not. Every request pays for the small call, and the failed ones pay for the big call on top.

Call the share of requests that get through the check the pass rate, p. Call the small model's cost per request divided by the big model's the price ratio, r. Then the cascade costs r + (1 − p) of what the strong model alone would cost, and your saving is p − r. The cascade breaks even when the pass rate equals the price ratio, and loses money below it.

Five pairs, one support ticket

To get real ratios I took one task: turn a customer email into a JSON ticket with six fields. The instructions and the email come to 346 input tokens and the reply to 65, counted with tiktoken, OpenAI's open-source token counter, using o200k_base, the encoding of current GPT models. List prices come from the catalog of the LLM gateway I work on, snapshot of 19 October 2026. I didn't measure pass rates: 70% in the last column is a round number for illustration.

Small → strong	List price in / out, $ per 1M	Break-even pass rate	Bill change at 70% pass
claude-haiku-5.5 → claude-sonnet-5.5	0.10 / 0.50 → 2 / 10	5%	−65%
gpt-5.6-mini → gpt-5.6	0.20 / 1.20 → 2 / 12	10%	−60%
gemini-3.8-flash → gemini-3.1-pro	0.75 / 3.75 → 2 / 12	34%	−36%
claude-sonnet-5.5 → claude-opus-5.5	2 / 10 → 4 / 20	50%	−20%
gemini-3.5-flash → gemini-3.1-pro	1.50 / 9 → 2 / 12	75%	+5%

The model's name tells you nothing about the ratio. Two Gemini "flash" models in front of the same 3.1 Pro land at 34% and 75%. Haiku in front of Sonnet pays off above a 5% pass rate, while Sonnet in front of Opus has to handle half the work just to break even. The 3.8 Flash row is the only one where the task shape matters: its input costs 37.5% of Pro's and its output 31%, so the ratio slides between those two numbers.

In dollars, at 100,000 tickets a month, gpt-5.6 alone costs $147 and the mini-first cascade $59. The Gemini 3.5 Flash cascade costs $155 against $147 for 3.1 Pro alone. You pay extra for an extra moving part.

Measure two numbers before you build it

Use what your calls actually cost, not list prices. The ratio in the table assumes both models write the same number of tokens. If the small model writes five times more, gpt-5.6-mini's ratio climbs from 10% to 31%. If it's a "thinking" model that spends 1,000 reasoning tokens, hidden notes billed as output, on this short ticket while the big one answers straight away, the ratio hits 92% and the cascade almost never pays. So run 50 real requests through the small model, check every reply, and plug in the average token counts the API returned:

# Averages from your own calls, and the prices you pay ($ per 1M tokens)
SMALL = {"in": 346, "out": 65, "price_in": 0.20, "price_out": 1.20}   # gpt-5.6-mini
BIG = {"in": 346, "out": 65, "price_in": 2.00, "price_out": 12.00}    # gpt-5.6
PASSED, TRIED = 35, 50   # small-model replies that passed your check

def cost(m):
    return (m["in"] * m["price_in"] + m["out"] * m["price_out"]) / 1e6

r = cost(SMALL) / cost(BIG)
p = PASSED / TRIED
print(f"break-even pass rate: {r:.0%}, yours: {p:.0%}")
print(f"cascade vs strong model only: {r + 1 - p:.0%} of the bill")

It prints break-even pass rate: 10%, yours: 70%, then cascade vs strong model only: 40% of the bill.

The check has to be automatic, or there's nothing to escalate on. For structured output it costs nothing: parse the JSON and compare the keys. Return which model answered and log it. In production, that log is your pass rate.

import json

KEYS = {"order_id", "product", "issue", "wants_refund", "urgency", "summary"}

def passes(reply):
    try:
        data = json.loads(reply)
    except ValueError:
        return False
    return isinstance(data, dict) and data.keys() == KEYS

def cascade(prompt, ask_small, ask_big):
    reply = ask_small(prompt)
    if passes(reply):
        return reply, "small"
    return ask_big(prompt), "big"

small = lambda p: 'Sure! Here is the JSON: {"order_id": "A-48213"}'
big = lambda p: json.dumps(dict.fromkeys(KEYS))
print(cascade("...", small, big)[1])

The two lambdas stand in for real API calls, so the snippet runs as is. The fake small model chats before its JSON, the parse fails, and it prints big.

What this doesn't solve

Failed requests wait for two calls, so at a 70% pass rate three tickets in ten get slower, not faster. A wrong answer that passes the check is never escalated: a key check catches broken JSON, not a wrong order number. Free-text answers have no cheap check, and grading them with a third model adds its cost to every request. tiktoken is OpenAI's tokenizer, so Claude and Gemini counts will differ.

Prices below list don't fix this and can make it worse. The gateway I mentioned sells all eight models under list, but not at the same discount: Gemini 3.1 Pro gets a deeper cut than 3.5 Flash, so at our prices that pair's break-even moves from 75% to 85%. Recompute the ratio at the prices you actually pay.

The takeaway

A cheap first try is never free. Divide what the two calls cost, measure how often the small model passes, and build the cascade only if the pass rate sits well above that ratio.

The model fixed one character and billed you for retyping all 497 lines

Andrey Altrouter — Sun, 25 Oct 2026 12:00:01 +0000

A typical first AI coding script reads a file, sends it with "fix this bug", and writes the reply back over the original. It works. But for a one-character fix in a 497-line file, the model writes back 4,495 tokens, the whole file again, and you pay for every unchanged line it retypes just to deliver one changed character.

The fix is one character, the reply is the whole file

You pay per token, a chunk of text the model reads or writes, roughly four characters of English or code. Prices are quoted per million tokens and split in two: input tokens are everything you send, output tokens are everything the model writes back. Output is the expensive side. On most current models it costs 5–6 times as much as input.

A whole-file request puts your file on both sides of the bill. The model reads it once as input, then writes it again as output, unchanged lines included. The fix you actually wanted is a rounding error inside that second number.

That gives a rule of thumb you can work out in your head. When the model has to retype the file, the edit costs close to 1 + (output price ÷ input price) times what it costs the model to read the file. If output is 6 times the input price, that's nearly 7×, and the bigger the file, the closer you get to that ceiling.

One off-by-one bug, five models

To put numbers on it, I took a real file: textwrap.py from Python's standard library, 497 lines and 4,491 tokens. I counted them with tiktoken, OpenAI's open-source token counter, using o200k_base, the encoding of current GPT models. I broke one line, turning <= into < in the check that decides whether a word still fits, and priced the fix two ways. If the model returns the whole corrected file, that's 4,495 output tokens. If it returns a SEARCH/REPLACE block, meaning the lines to find and what they should become, that's 51.

All prices are list prices, the per-token prices vendors publish. They come from the catalog of the LLM gateway I work on, snapshot of 17.10.2026.

Model	List price in / out, $ per 1M	Whole file back	Change only	Ratio
gpt-5.6-mini	0.20 / 1.20	$0.0063	$0.0010	6.5×
gpt-5.6	2 / 12	$0.0630	$0.0098	6.5×
claude-sonnet-5.5	2 / 10	$0.0540	$0.0097	5.6×
deepseek-v4.1-flash	0.30 / 1.20	$0.0068	$0.0014	4.7×
grok-4.3	1.25 / 2.50	$0.0169	$0.0058	2.9×

On gpt-5.6, 86% of the whole-file bill is the file being typed back. Almost all of the change-only cost is the model reading the file. The ratio doesn't follow the price level: gpt-5.6-mini is ten times cheaper and lands at the same 6.5×. It follows the gap between output and input prices, which is why grok-4.3, where output costs only twice as much as input, gains least. File size matters less than you'd think. With the same block of about 50 tokens, getopt.py (215 lines) comes out at 5.8× and argparse.py (2,676 lines) at 6.9×.

For a script that makes 200 edits a day on gpt-5.6, that's $378 a month against $59.

Ask for the change, not the file

Add this to your instructions. It costs 53 input tokens per call:

Return only the change, never the whole file. Use one block per change:
<<<<<<< SEARCH
(the exact lines to change, copied from the file, plus one unchanged line above them)
=======
(the same lines as they should read after the change)
>>>>>>> REPLACE

The unchanged line goes in both halves. It's there to make the match unique, because a single line like return None may appear a dozen times in a file. Then apply the reply yourself, and refuse anything you can't match exactly once:

import re
BLOCK = re.compile(r"<<<<<<< SEARCH\n(.*?)\n=======\n(.*?)\n>>>>>>> REPLACE", re.S)

def apply_edits(source, reply):
    blocks = BLOCK.findall(reply)
    if not blocks:
        raise ValueError("no SEARCH/REPLACE block in the reply")
    for old, new in blocks:
        if source.count(old) != 1:
            raise ValueError(f"search text found {source.count(old)} times")
        source = source.replace(old, new)
    return source

src = "def area(w, h):\n    # rectangle\n    return w + h\n"
reply = "<<<<<<< SEARCH\n    # rectangle\n    return w + h\n=======\n    # rectangle\n    return w * h\n>>>>>>> REPLACE"
print(apply_edits(src, reply))
try:
    apply_edits(src, reply.replace("w + h", "w - h", 1))
except ValueError as e:
    print("refused:", e)

It prints the fixed function, then refused: search text found 0 times. Zero means the model misquoted your code, and more than one means the match is ambiguous. Guessing in either case could corrupt the file, so the function stops.

When an edit is refused, ask again for the whole file. That fallback costs one extra call, but a failed change-only call costs little more than reading the file. So on gpt-5.6 the format stays cheaper until about 85% of your edits need the fallback.

What this doesn't solve

The input side doesn't shrink: the model still reads all 4,491 tokens. Cutting that means sending only the relevant function, a different trade-off. Reasoning tokens, the notes a "thinking" model writes to itself before it answers, are billed as output in both modes. With 2,000 of them, gpt-5.6's 6.5× shrinks to 2.6×. I didn't measure how often models misquote the search text. That depends on the model and the file, so count your refusals. Claude and Gemini also split text into tokens differently from tiktoken, so the counts will shift. The ratio follows the prices.

A cheaper price per token doesn't do this job either. The gateway I mentioned sells these models 10–29% below list. But a discount takes the same share off both modes: at our price, gpt-5.6's whole-file edit is $0.054, still 5.5 times the change-only edit at full list price.

The takeaway

The fix was one character. Ask for the change, apply it with an exact-match check, and you pay once to have the file read instead of paying output price to have it typed back.

Your chatbot's average call costs $0.006 — one pasted file can make it $2.03

Andrey Altrouter — Sat, 24 Oct 2026 11:00:02 +0000

I priced a typical support-bot question on Claude Sonnet 5.5 at list price: $0.006. The same endpoint, with the same key and the same code, will accept a request that costs $2.03, as much as 339 typical questions at once. Nothing in the API asks whether you meant it. All it takes is one user pasting a big file into your text box.

Every request has a price ceiling

You pay per token, a chunk of text about three-quarters of an English word. Prices are quoted per million tokens, separately for input (everything you send: instructions, chat history, the user's message) and output (what the model writes back). The context window is the most tokens one request may hold. max_tokens is the request field that caps the length of the reply.

Put them together and you get the most a single request can cost: input tokens you let in × input price + max_tokens × output price.

If you don't limit the input, the first term is the whole context window. Nobody types a million tokens by hand, but three ordinary things get you there. A user pastes a log file into the chat box. A bug appends the same document to the prompt on every loop. A search step hands the model whole files instead of the relevant paragraphs. The API treats none of these as an error. If it fits the window, it's a valid request, and it's billed.

Five models, one full window

I compared a typical call (1,500 tokens in, 300 out: a system prompt, a short history and a question) with a call that fills the window and gets a 4,096-token reply. Context windows and list prices, meaning the vendors' own published per-token prices, come from the catalog of the LLM gateway I work on (snapshot of 16.10.2026), where prices sit 10–29% below list, which shrinks every dollar cell and leaves the ratio column untouched.

Model	Context window	Input, $/1M	Typical call	Full-window call	Ratio
gpt-5.6-mini	400K	$0.20	$0.0007	$0.08	127×
claude-sonnet-4-5	200K	$3	$0.0090	$0.65	72×
gpt-5.6	400K	$2	$0.0066	$0.84	127×
claude-sonnet-5.5	1M	$2	$0.0060	$2.03	339×
gpt-6-astra	1.05M	$10	$0.0300	$10.66	355×

The ratio doesn't follow the price. It follows the window. Across the whole catalog, every 200K model lands between 72× and 79×, and every 1M model between 258× and 375×. claude-haiku-5.5 at $0.10 per million and claude-fable-5.1 at $10 both come out at exactly 339×.

That's how an upgrade can cut your price per token and still raise your worst case. Moving from claude-sonnet-4-5 to claude-sonnet-5.5 makes input a third cheaper, and the typical call drops from $0.0090 to $0.0060. The full-window call goes from $0.65 to $2.03, because the window grew five times.

A megabyte of logs is half a million tokens

How much text fills the window depends on what's pasted, and the usual rule of thumb, four characters per token, only holds for prose. I counted with tiktoken, OpenAI's open-source tokenizer (the program that cuts text into tokens), using o200k_base, the encoding behind current GPT models.

95 KB of English articles from this blog came out at 24,434 tokens, 3% above the four-characters estimate. A synthetic one-megabyte server log came out at 529,646 tokens, 2.1 times the estimate. It's lines like 2026-10-14T09:23:18.178Z WARN req=c41f9dfd-… POST /health/66746 201 1647ms, and random hex IDs and long numbers split into many small tokens.

So one megabyte of logs is $1.06 of input on claude-sonnet-5.5 and $2.12 on claude-opus-5.5. On gpt-5.6 it doesn't fit at all: the 400K window is full at 0.76 MB, and the API answers with an error instead. The paste that costs you money is the one that just fits.

It also means a guard that counts characters lets through twice the tokens you planned for whenever someone pastes machine output.

Set the ceiling yourself, in dollars

Decide what one request may cost, and check it before sending: count the input tokens, add the reply ceiling, multiply by your model's prices, refuse anything above the limit. It needs pip install tiktoken.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # GPT-5.x tokenizer; others differ
PRICE_IN, PRICE_OUT = 2.00, 12.00          # $ per 1M tokens, gpt-5.6 list price
MAX_TOKENS = 1_000                         # send the same value as max_tokens
MAX_USD = 0.02                             # the most one request may cost

def check(messages):
    tokens_in = sum(len(enc.encode(m["content"])) + 4 for m in messages)
    usd = (tokens_in * PRICE_IN + MAX_TOKENS * PRICE_OUT) / 1e6
    if usd > MAX_USD:
        raise ValueError(f"refused: up to ${usd:.2f}, limit ${MAX_USD:.2f}")
    return usd

print(f"ok: up to ${check([{'role': 'user', 'content': 'Where is my order?'}]):.4f}")
paste = "2026-10-14T09:23:18.178Z WARN req=c41f9dfd-b484 POST /health 201\n" * 5_000
try:
    check([{"role": "user", "content": paste}])
except ValueError as e:
    print(e)

It prints ok: up to $0.0120, then refused: up to $0.32, limit $0.02. The 325 KB paste never reaches the API, so it costs nothing. The + 4 is a rough allowance for the few tokens of wrapping each message carries.

Look at where the short question's $0.012 comes from: almost all of it is max_tokens, 1,000 reply tokens at $12 per million. If your bot usually answers in 300 tokens, a lower max_tokens is the cheaper lever. At these settings the guard leaves room for 4,000 input tokens, about 2,700 words. If that's too tight, raise the limit. What matters is that it's a number you picked. Instead of refusing, you can also trim the paste to fit and tell the user.

What this doesn't solve

tiktoken is OpenAI's tokenizer. Claude and Gemini split text differently, so on those models treat the count as an estimate and keep a margin. A per-request ceiling also says nothing about how many requests arrive: a runaway loop stays under the limit every single time. That needs a daily cap. Caps have a gap of their own: a request's price is known only after the reply ends, so spend limits are usually checked before a request and charged after it. Ours works that way too, and the request that crosses the limit is still billed in full.

The takeaway

Until you set a lower one, the context window is the price ceiling of every request. Pick a dollar limit, count tokens rather than characters, and check before you send.

The "next version" of your LLM can cost 33% less — or 4x more

Andrey Altrouter — Fri, 23 Oct 2026 11:00:01 +0000

Upgrading an LLM looks like the cheapest change in your codebase. You replace "gemini-2.5-flash" with "gemini-3.5-flash" and redeploy. One string, no new library, no new API.

On your invoice it is not one string. On the same traffic, that edit makes the bill 4.1 times bigger. The same kind of edit on Claude Sonnet makes it a third smaller. The version number tells you nothing about which way the price will move.

What changes when you change the model string

Every request names a model by its model ID — the string like gpt-5.6 that you pass in the model field. You pay per token: a chunk of text, roughly three-quarters of an English word. Prices are quoted per million tokens and split in two: input tokens are everything you send (instructions, chat history, documents), output tokens are what the model writes back. Output typically costs 4 to 8 times more than input.

Each model ID has its own pair of prices. When a vendor ships a new version, it sets a new pair, and nothing obliges it to go down. Sometimes the new model is cheaper to run and the vendor passes that on. Sometimes the new model is positioned a tier higher and priced like it.

Your code doesn't notice either way. The request format is identical, the response looks the same, and the price change arrives silently at the end of the month.

Six upgrades, one workload

To compare like with like, I fixed one workload: a small app that sends 20 million input tokens and gets back 4 million output tokens a month, a 5:1 ratio. Then I priced it before and after a same-vendor upgrade, using the vendors' list prices (their own published per-token prices) from our catalog snapshot of 16.10.2026.

Upgrade	Old, in / out per 1M	New, in / out per 1M	Monthly bill	Change
claude-sonnet-4-5 → claude-sonnet-5.5	$3 / $15	$2 / $10	$120 → $80	−33%
claude-opus-4-5 → claude-opus-5.5	$5 / $25	$4 / $20	$200 → $160	−20%
gpt-4o → gpt-5.6	$2.50 / $10	$2 / $12	$90 → $88	−2%
gpt-4o-mini → gpt-5.6-mini	$0.15 / $0.60	$0.20 / $1.20	$5.40 → $8.80	+63%
gemini-2.5-flash → gemini-3.5-flash	$0.30 / $2.50	$1.50 / $9	$16 → $66	+313%

Same direction in the name, opposite directions on the bill. Both Anthropic upgrades got cheaper. OpenAI's flagship stayed flat, while its small model doubled its output price. Google's Flash line went up the most: five times on input, 3.6 times on output. A sixth pair I checked, gemini-2.5-pro → gemini-3.1-pro, comes out +35% on the same workload.

The bottom two rows are the trap for beginners. Small, cheap models are the ones tutorials hardcode, and "cheap" in an older tutorial is a statement about an older price list.

When the answer depends on your traffic

The gpt-4o → gpt-5.6 row deserves a second look. Input got cheaper by $0.50 per million, output got pricier by $2. Which one wins depends on your mix, and the break-even is exactly 4 input tokens per output token. At 4:1 the bill doesn't move. At 12:1, an app that stuffs documents into every prompt, the upgrade saves 10%. At 2:1, an app that writes long answers to short questions, it costs 6.7% more.

There is a second multiplier the price list doesn't show. Newer models often write more for the same task: longer answers, or reasoning tokens, hidden "thinking" text the model produces before answering and bills as output even though you never see it. The price per token can fall while the tokens per task grow.

For the Sonnet upgrade I computed the margin: on this workload, claude-sonnet-5.5 stays cheaper than claude-sonnet-4-5 until its output doubles. That sounds like plenty of room. With reasoning switched on, it isn't always.

Check before you bump

Before changing a model ID, put both price pairs and your real monthly token counts into a few lines of arithmetic. Your counts are in the usage field of every API response, or on your provider's billing dashboard.

def monthly(in_m, out_m, price_in, price_out):
    return in_m * price_in + out_m * price_out

# your traffic, millions of tokens per month
IN_M, OUT_M = 20, 4
# expected output growth on the new model: 1.0 = same, 1.5 = 50% more
OUT_GROWTH = 1.5

old = monthly(IN_M, OUT_M, 0.30, 2.50)               # gemini-2.5-flash
new = monthly(IN_M, OUT_M * OUT_GROWTH, 1.50, 9.00)  # gemini-3.5-flash
print(f"${old:.2f} -> ${new:.2f}  ({(new - old) / old:+.0%})")

This prints $16.00 -> $84.00 (+425%). The 1.5 is a guess, so replace it with a measurement: send the same few hundred real requests to both models and compare the output token totals. That measured ratio is the number worth trusting.

Upgrades usually come with a good reason, like better answers or a deprecation notice for the old model. The point isn't to avoid them. It's to know what the upgrade costs before the invoice tells you.

What this doesn't solve

This compares price, not quality. A pricier model that gets the task right on the first try can beat a cheap one you have to call three times, and I haven't measured that here. The table also uses list prices. Resellers price below list, and not always by the same percentage across versions: in our own catalog claude-opus-4-5 is 25% off and claude-opus-5.5 is 15% off, so at our prices that upgrade saves 9%, not 20%. Compare the prices you actually pay.

The takeaway

A new model ID is a new price list. Before you change the string, multiply your own token counts by both price pairs, then measure how many output tokens the new model actually writes.