How One Bad Chunk Can Triple Your LLM Bill
After Episode 2, I trusted my extraction pipeline. Clean text, tables intact, headers stripped out. I felt like I'd earned the right to stop worrying about the early stages and move on.
So I moved on to chunking, expecting it to be the boring part. Split the text, pick a size, done.
The first version I shipped used a small chunk size on purpose. The reasoning felt sound: smaller chunks mean each one is more focused, retrieval should be more precise, nothing gets buried under irrelevant text. Felt like the safe choice.
It didn't feel small until I ran it across a real batch of documents. A handful of test PDFs turning into a few hundred chunks looks completely fine on a laptop. The same chunk size, run across a production-sized document set, turned into far more chunks than I'd actually sat down and calculated in advance — and the ingestion job I expected to finish in a few minutes was still running an hour later, CPU pinned the whole time, one embedding call after another, after another.
That was the first real cost I hadn't planned for. Not a wrong answer. Not a dollar figure. Just time and CPU quietly disappearing into a chunking decision I'd made without thinking about scale.
So I did what felt like the obvious fix: increase the chunk size. Fewer, bigger chunks. Ingestion sped back up, CPU load dropped, the embedding job finished in a reasonable window again.
Except now a different cost showed up downstream, where I wasn't looking. Retrieval started pulling in noticeably more text per chunk than a query actually needed. The LLM was reading through extra surrounding paragraphs just to get to the one sentence that mattered. Response latency crept up. Prompts got heavier. I'd fixed the ingestion-time cost and quietly created a per-query cost that was going to be paid on every single request, forever, not just once during ingestion.
That's the part that actually changed how I think about chunking. It isn't one setting with one cost. Go too small, and you pay in CPU and time during ingestion, before a single user ever asks a question. Go too big, and you pay in latency and token overfeed on every query, long after ingestion is finished. There wasn't a "perfect" number waiting to be discovered — there was a workable middle ground, found by actually watching where the time and compute were going, not by copying a number off a tutorial.
The Idea This Whole Article Is About
A chunk boundary isn't a formatting choice. It's a cost decision — and "cost" here doesn't just mean dollars.
It means CPU. RAM. Storage. Vector DB size. Network transfer. API calls. Engineering time. LLM tokens. Latency. And eventually, user trust.
Chunk size looks like a single number you set once — 500 tokens, 1000 tokens — and forget. But that one number quietly touches almost every resource your system consumes, on every single request, for as long as that chunking decision stays in production.
At first I thought chunk size was just another configuration value. Pick 500 or 1000 tokens and move on.
Then I realized every chunk size creates a different bill. One increases retrieval failures. Another increases token usage. Another inflates storage. Another increases embedding count. None of them are free — they just charge different resources.
So instead of asking "how does chunking work," I started asking a different question for every stage of the pipeline: what does this chunk size make more expensive here?
Cost 1 — Embedding
Smaller chunks
│
▼
More chunks per document
│
▼
More embedding calls
│
▼
More CPU cycles, more wall-clock time spent embedding
│
▼
Ingestion job that should take minutes takes an hour
This is exactly what happened in my ingestion run. Embedding a single chunk is cheap. But cheap-per-call times far more calls than I'd planned for is what turned a quick batch job into CPU sitting pinned for an hour, with nothing to show for it except a longer wait before the system was even usable. It's the smallest cost per unit, but it's the first domino — every other cost on this list traces back to how many chunks you decided to create in the first place.
Cost 2 — Storage
More chunks don't just mean more embeddings. Each chunk also drags its own copy of metadata along with it.
Imagine every chunk stores something like:
Document ID
Title
Author
Department
Permissions
If a document gets split into a small number of large chunks, that metadata gets duplicated a small number of times. If the same document gets split into a large number of tiny chunks, that same metadata block now gets duplicated once per chunk — potentially hundreds of thousands of times across a large knowledge base.
Smaller chunks
│
▼
More chunk records
│
▼
Metadata duplicated once per chunk
│
▼
Storage grows faster than the actual content did
Nobody thinks of metadata as a chunking cost. But it scales directly with chunk count, not with document size — which means an aggressive chunking strategy can inflate storage even when the underlying documents haven't changed at all.
Cost 3 — Vector Database
More vectors means a bigger index. A bigger index isn't just "more storage" — it's more RAM and more replica capacity needed to keep that index searchable in milliseconds, running whether or not anyone's actually querying it right now.
More chunks
│
▼
More vectors
│
▼
Larger index
│
▼
More RAM / more nodes to keep it fast
│
▼
Higher standing infrastructure cost (Pinecone, Qdrant, pgvector, whatever you run)
This is a recurring cost, not a one-time one — unlike embedding, which you pay once, the vector database keeps charging you for existing every single day, proportional to how many chunks you decided to create.
Cost 4 — Retrieval
A bigger index doesn't just cost more to store. It costs more to search.
Searching 500k vectors
│
▼
Fast, cheap similarity search
Searching 2 million vectors (same content, smaller chunks)
│
▼
More compute per query
│
▼
Potentially slower response times
And once a chunk is retrieved, it has to travel — from the vector DB, across the network, into your application, into the prompt. Sending 3 well-chosen chunks is a small payload. Sending 10 chunks because your chunking strategy fragmented the answer across more pieces is a bigger payload, on every single query, forever. It's a small cost per request, but it's a cost that scales with chunk count, not with how many people are actually asking questions.
Cost 5 — The LLM
This is the one most people already suspect, and it's real: more or larger chunks in the prompt means more tokens read on every call.
Bigger prompt
│
▼
More input tokens
│
▼
Higher inference cost, per query, every time
The part that's easy to miss: this cost doesn't scale with how much useful information is in the prompt. It scales with how much text is in the prompt. A chunking strategy that pulls in extra, irrelevant surrounding text pays this bill just as much as one that pulls in exactly the right paragraph — the LLM doesn't get a discount for reading filler.
Putting a Number on It
Everything above is easier to feel with actual numbers next to it — so here's a small worked example. These figures are illustrative, not measured from a real system — the point is the shape of the math, not the exact digits.
Example
100 documents
│
▼
300-token chunks → ~8,000 chunks → ~8,000 embeddings
versus
800-token chunks → ~3,000 chunks → ~3,000 embeddings
Fewer, bigger chunks mean fewer embeddings — that part looks like a win. But watch what happens the moment a real query comes in and retrieval pulls back the usual top-3 chunks:
3 chunks × 300 tokens = 900 prompt tokens
3 chunks × 800 tokens = 2,400 prompt tokens
Retrieval got simpler. The LLM now reads almost 3× more text on every single request — not once, but on every question, forever, for as long as that chunk size stays in production.
Now stack that on top of Cost 6 below. If the bigger chunks also happen to bury the answer under more irrelevant surrounding text, and the first response is incomplete enough that the user has to ask again, you're not just paying 3× the tokens once — you're paying that inflated prompt size again on the retry. That's how a single sizing decision, made once at ingestion, turns into a bill that's tripled by the time a real conversation is done. This is the concrete shape of the claim in the title — not a one-off freak accident, just what the token math does when it compounds with a retry.
Cost 6 — The Retry
Bad sizing isn't the only way chunking creates this cost. A chunk boundary landing mid-idea does it too — I hit this on a different document, where a policy explanation got cut clean in half by an unlucky boundary.
Chunk boundary cuts an idea in half
│
▼
Query matches only part of it
│
▼
LLM answers confidently, using incomplete context
│
▼
Answer is wrong, but doesn't look wrong
│
▼
User pushes back — 2nd LLM call
│
▼
User rephrases — 3rd LLM call
This is the other way the title's claim plays out in practice. It's also the hardest one to notice, because nothing in your logs says "this failed." The system answered. It just answered incompletely, and the actual cost shows up three questions later, spread across the LLM bill, disguised as normal usage.
Heavy chunk overlap makes a version of this worse in the other direction: near-duplicate chunks fill retrieval slots that should've gone to genuinely different information, so the LLM reads redundant text and still misses what it needed — same outcome, same retry, different cause.
Cost 7 — Rechunking (the one nobody plans for)
Here's the cost that almost never makes it into a chunking article.
Say you launched with 300-token chunks. Six months later, after watching real queries fail, you figure out 800-token chunks would've worked better for your document types.
You can't just change a config value and redeploy. Chunking isn't a setting — it's baked into every vector that already exists.
Decide chunk size should change
│
▼
Delete existing vectors
│
▼
Re-chunk every document from scratch
│
▼
Re-embed every new chunk
│
▼
Rebuild the vector index
│
▼
Rewrite metadata for every chunk
│
▼
Engineering time to test and validate the migration
│
▼
All of that, before a single user question benefits from it
Once a document is chunked, every later stage depends on that specific cut. Changing chunk size later isn't tweaking a variable — it's rebuilding your knowledge base from the ingestion stage forward. This is also exactly why getting chunk size closer to right the first time matters more than it seems to at launch: the cost of being wrong doesn't stay small, it compounds until someone has to pay to undo it.
Cost Isn't Just a Dollar Amount
It's worth saying plainly, because it's easy to read a "where does the money go" series and assume dollars are the only unit that matters.
They aren't. Every section above was really describing the same chunk-size decision spending a different resource:
- CPU / GPU — embedding and re-embedding cycles
- RAM — keeping a larger index searchable
- Storage — vectors and duplicated metadata
- Vector DB infrastructure — nodes, replicas, index size
- Network — chunks transferred per query
- API calls — embedding calls, LLM calls, retries
- Engineering time — tuning, debugging, rechunking migrations
- Latency — larger indexes and bigger prompts both slow things down
- User trust — the resource that doesn't show up on any invoice, and is the hardest one to earn back once a wrong answer spends it
A dollar figure is just where several of these eventually get converted so someone can put them on a budget. The actual damage happens earlier, resource by resource.
Bringing It Back to One Sentence
Chunking doesn't create one cost. It's a single decision that quietly multiplies almost every other cost in the pipeline — CPU time during ingestion if you go too small, latency and token overfeed on every query if you go too big, plus storage, vector DB size, retrieval compute, network transfer, retries, and eventually the cost of rebuilding all of it when you get it wrong.
I started this series thinking embeddings were where RAG got expensive. By Episode 3, the pattern is impossible to ignore: the earlier a decision happens in the pipeline, the more downstream resources it quietly taxes — and chunking sits closer to the start than almost anything else. Getting it "production fine" isn't about finding a perfect number. It's about watching where the CPU and the time actually go, and moving the number until both ends stop hurting.
Coming up in Episode 5
Chunking taught me that how I slice knowledge affects nearly every resource downstream. But there's a layer I've been ignoring completely: the metadata attached to each chunk.
What happens when your chunks are perfect, but retrieval has no idea which ones actually matter?
Less noise, more action. Let's dig.
Top comments (0)