Your LLM bill has two sides. Build the ledger that shows both.

Every RAG cost estimate starts the same way: input tokens equal top_k times chunk size, plus some overhead. Most of them stop there too. Then the invoice arrives, it is three times the estimate, and the team spends a sprint tuning chunk sizes while the actual money leaks somewhere that formula never touches.

The bill for an LLM system is set by total input and output tokens, summed across every call a query triggers. Not per call. Not input only. Every call, both directions. If you only remember one sentence from this post, that is the one.

Why chunk math lies

Four leaks, in the order I usually find them in real pipelines:

Calls you forgot exist. A query condenser rewriting the user question. A silent SDK retry. A JSON repair call after a malformed response. An agent loop that took three turns instead of one. Each is a full-priced API call that no chunk formula predicts, and some frameworks make them without telling you.

Scaffolding re-sent on every call. System prompt, tool schemas, formatting instructions, few-shot examples. This fixed overhead rides along on every single call, and in multi-call pipelines it often outweighs the retrieved chunks you are carefully tuning.

The output side, billed at a premium. Output tokens cost several times input on every major provider's list price, five times on current Claude models. A dashboard that tracks input tokens is watching the cheap half of the transaction.

Multi-call synthesis re-billing its own draft. Refine-style chains regenerate the full running answer at every step. Only the final draft is delivered, but all N drafts are billed at output rates, and each discarded draft is re-sent as input to the next call. The answer component of your cost scales with N, and no chunk tuning touches it. Prompt caching does not rescue it either: the growing answer sits mid-prompt, which breaks the prefix match.

Stop counting tokens. Start reading them.

The fix is not a better estimate. It is refusing to estimate: every provider already returns exact usage on every response, so the whole job is logging what the API tells you and adding it up per query.

Where you hook matters more than it looks:

• The SDK call site is the reliable choke point. A thin wrapper around each create() sees server-reported usage for every call you make, streaming or not.
• Framework callbacks are leakier. Some read server-reported usage properly and propagate through nested calls (LangChain's usage metadata callback does). Others estimate with a local tokenizer instead of reading the wire, LlamaIndex's TokenCountingHandler being the documented example. And none of them see calls made outside the framework.
• HTTP-layer hooks have a streaming blind spot. An httpx response hook fires when headers arrive, before the body. For non-streaming calls you can read the body inside the hook; for SSE streams the usage arrives at the end of a body the hook cannot safely consume. If you want wire-level capture for streams, you need a logging proxy or a transport wrapper that tees the stream.

For most codebases the call-site wrapper is one afternoon of work and covers everything that matters.

The ledger

Columns that earn their keep, learned from what actually gets queried later:

column	why it exists
query_id	ties every call in one user request together; propagate it with a contextvar
call_index	0, 1, 2... within the query; the column that exposes hidden calls
model	mixed-model pipelines bill at mixed rates
input_tokens, output_tokens	the two sides of the bill, server-reported
cache_read, cache_write	cached input bills at its own rates; on Anthropic these are two separate fields billed differently
stop_reason	a truncated response often triggers a retry you will want to find
latency_ms	free to record, and cost and latency investigations are usually the same investigation
stage	"condense", "retrieve-answer", "repair"; the label that turns rows into a story

A minimal implementation, SQLite plus a contextvar:

import sqlite3, time, uuid
from contextvars import ContextVar

query_id: ContextVar[str] = ContextVar("query_id", default="")

db = sqlite3.connect("tokens.db")
db.execute("""CREATE TABLE IF NOT EXISTS calls(
    query_id TEXT, call_index INT, model TEXT,
    input_tokens INT, output_tokens INT,
    cache_read INT, cache_write INT,
    stop_reason TEXT, latency_ms INT, stage TEXT)""")

def record(model, usage, stop_reason, latency_ms, stage=""):
    qid = query_id.get() or str(uuid.uuid4())
    idx = db.execute(
        "SELECT COUNT(*) FROM calls WHERE query_id=?", (qid,)
    ).fetchone()[0]
    db.execute("INSERT INTO calls VALUES (?,?,?,?,?,?,?,?,?,?)",
        (qid, idx, model,
         usage["input"], usage["output"],
         usage.get("cache_read", 0), usage.get("cache_write", 0),
         stop_reason, latency_ms, stage))
    db.commit()

Wrapping a non-streaming Anthropic call:

import anthropic, time

client = anthropic.Anthropic()

def chat(stage="", **kwargs):
    t0 = time.monotonic()
    resp = client.messages.create(**kwargs)
    u = resp.usage
    record(
        model=resp.model,
        usage={
            "input": u.input_tokens,
            "output": u.output_tokens,
            "cache_read": getattr(u, "cache_read_input_tokens", 0) or 0,
            "cache_write": getattr(u, "cache_creation_input_tokens", 0) or 0,
        },
        stop_reason=resp.stop_reason,
        latency_ms=int((time.monotonic() - t0) * 1000),
        stage=stage,
    )
    return resp

Set the contextvar once at the top of each user request (query_id.set(str(uuid.uuid4()))), pass a stage label at each call site, and the ledger fills itself.

The streaming gotchas

Streaming is where naive capture quietly loses data, and the two big providers fail in different directions:

OpenAI only emits usage on a streamed call if you ask for it: pass stream_options={"include_usage": True}. The usage then arrives on the final chunk, with every earlier chunk carrying usage=None. Forget the flag and the ledger records nothing while the meter runs.

Anthropic splits usage across the stream: message_start carries the input side, and the output count arrives via message_delta near the end. If you use the SDK's streaming helper, the accumulated final message carries complete usage, so stream.get_final_message().usage is the easy path.

Field names do not agree across providers. prompt_tokens versus input_tokens, cached tokens under prompt_tokens_details.cached_tokens on one API and cache_read_input_tokens on another. And some self-hosted OpenAI-compatible servers omit usage entirely. Normalize into your own schema at write time, not at query time.

What the first week shows you

Having turned this on in production pipelines, the surprises arrive in a reliable order.

First surprise: call count. A rewrite step someone added in March, a retry that fires on every truncated response, an agent loop budgeted for one turn that averages 2.4. GROUP BY query_id and the queries with eight rows instead of two are your bill.

Second: the scaffolding. The fixed prompt overhead, multiplied by the call count you just discovered, frequently outweighs the retrieved context. People tune the 2,000 variable tokens and ignore the 3,500 fixed ones riding on every call.

Third, and only third: chunk size and top_k, the things everyone tunes first.

The ledger also makes redundancy visible. If your corpus has near-duplicate documents, they show up as repeating, near-identical input deltas across queries: you are paying to send the model the same paragraphs again and again, and the fix (dedup before the context window) is a quality fix that happens to cut the bill.

The whole analysis, honestly, is one query:

SELECT query_id, COUNT(*) AS calls,
       SUM(input_tokens) AS input,
       SUM(output_tokens) AS output,
       SUM(input_tokens + 5 * output_tokens) AS cost_proxy
FROM calls
GROUP BY query_id
ORDER BY cost_proxy DESC LIMIT 20;

The 5 is a list-price output-to-input ratio; swap in your provider's. The top 20 rows of that query are worth more than any cost-optimization blog post, including this one.

Sid Probstein made the argument from the architecture side this week, comparing what RAG stacks actually send across frameworks, and his closing line is the right summary of the whole subject: that number, not top_k, is your bill.