How many tokens does your RAG stack actually send to the LLM?

The token bill for a RAG system is not set by your vector database. It's set one step later, by how you assemble the context you hand the LLM. Retrieval finds candidates; assembly decides how many of them, how much of each, and across how many LLM calls. That's where the money is spent.

So "vector DB vs. framework X vs. SWIRL" is the wrong axis. The real comparison is between context-assembly strategies. Here's an honest, source-checked look at how many tokens each common approach sends, and where SWIRL 5 actually costs less.

The honest headline first

SWIRL doesn't win by using smaller chunks. Anyone can lower top_k. On a single lean query, a minimal config like LlamaIndex's default (top_k=2) sends fewer raw tokens than SWIRL. If someone tells you SWIRL "always uses the fewest tokens," a technical evaluator will disprove it in five minutes.

SWIRL's advantage is structural, and it shows up exactly where enterprise content lives, in corpora full of document versions:

1. It never spends tokens on duplicate or superseded versions. Stock top-k returns whatever is nearest in embedding space, which in a versioned corpus means several near-identical copies. SWIRL collapses them to one canonical document before the LLM sees them.
2. It always answers in one bounded call. The multi-document synthesis modes people reach for when they want quality (LangChain refine/map_reduce, LlamaIndex refine/tree_summarize) multiply LLM calls, and refine multiplies tokens super-linearly. SWIRL never pays that tax.

What the common defaults actually do

Naive vector-DB RAG (Pinecone / Weaviate / Qdrant tutorials)

The pattern is: embed the query → retrieve top-k chunks → stuff them all into one prompt. Typical documented defaults:

• Chunk size ~512 tokens (the common "start here"; the band is 256-1024)
• Chunk overlap 10-20%
• top_k 3-5
• No de-duplication. None of the vendor quickstarts add a dedup or diversity step. Overlap alone guarantees adjacent chunks share text, and multiple versions of a document sit in the same embedding neighborhood, so top-k routinely returns redundant context and the pipeline sends all of it.

Input tokens ≈ k × chunk_tokens + overhead. At k=5 and ~1,000-token chunks, that's ~5,000 tokens, a large fraction of it redundant.

Sources: Pinecone, Weaviate, Qdrant.

LangChain

Verified from source:

• Retriever default k=4 (similarity_search(k=4) in vectorstores/base.py)
• RecursiveCharacterTextSplitter default 4,000 chars / 200 overlap (≈1,000 tokens) (text_splitters/base.py)

The combine-documents chains differ enormously in cost:

Chain	LLM calls (N docs)	Token behavior
stuff	1	all docs in one prompt
map_reduce	N + ≥1	one call per doc, then reduce
refine	N, sequential	re-sends the growing answer each step → super-linear tokens, no parallelism
map_rerank	N	one doc per call

refine is the trap: each step re-transmits the accumulating answer plus the next document, so the running answer is re-sent N-1 times and grows as it goes.

LlamaIndex

Verified from source (llama-index-core):

• chunk_size 1,024 tokens, similarity_top_k 2, default response mode compact (constants.py, factory.py)
• No default de-duplication. Node postprocessors are opt-in and the default similarity cutoff is off, so two versions of a doc in top-k both go to the LLM.

Response modes (docs):

Mode	LLM calls	Token behavior
compact (default)	~1	packs nodes into as few prompts as fit
refine	N	one call per node, re-sends evolving answer
tree_summarize	>1, recursive	summarize groups, then summaries-of-summaries
accumulate	N	query each node separately

compact with top_k=2 is genuinely lean, but that's a recall trade-off, and it still sends duplicate versions.

What SWIRL 5 does (verified in source)

SWIRL treats the LLM prompt as a hard budget and fills it deliberately:

Mechanism	Behavior
Hard prompt budget	RAG prompt capped at ~3,000 tokens (SWIRL_RAG_TOK_DEFAULT); assembly stops when full
Source cap + relevance gate	≤ 10 sources, only those scoring ≥ 0.8 are eligible
Scored semantic chunks	a BM25 topic-matcher scores spans within each source; on overflow SWIRL narrows to those scored spans, not the whole chunk
Truncation	per-source token-by-token truncation to fit the budget
Markup	relevant spans wrapped in <SW-IMPORTANT>…</SW-IMPORTANT> plus a compact per-source metadata header
De-dup before the LLM	version-cluster alternates dropped; if a doc is pinned, only the canonical is kept, so N versions collapse to 1
Single call	one stuff-style synthesis call (worst case +1 JSON repair)

Net: ~3,000 input tokens, one call, zero redundant-version tokens, a predictable ceiling that doesn't grow with document size or corpus size.

Putting numbers on it

Scenario: a query over a corpus where the relevant policy exists in 5 versions, plus 3 other relevant documents, a realistic enterprise shape. Total LLM input tokens per query (summed across calls):

Reading it honestly:

• vs. a deliberately minimal stuff config (LlamaIndex compact, top_k=2): SWIRL is comparable per call, but that config buys its low count with poor recall and still ships duplicate versions.
• vs. typical stuff RAG (k=5, ~1k chunks): SWIRL is ~40-45% fewer input tokens, and it removes the redundant-version half entirely.
• vs. the quality-oriented multi-call modes (refine, tree_summarize, map_reduce): SWIRL is a 2-3× reduction in input tokens, and a larger reduction in output tokens, since those modes generate once per call.

Where the advantage is real, and where it isn't

• Real and hard to get off-the-shelf: cross-version de-duplication. None of these frameworks do it by default. In a versioned corpus it's the difference-maker, and it compounds: the more versions, the more SWIRL saves.
• Real: one bounded call vs. N-call synthesis; a predictable cost ceiling.
• Honest caveat: a hand-tuned vanilla RAG (low k, a good reranker, small chunks, stuff, plus your own dedup layer) can match SWIRL's per-call token count. SWIRL's value is delivering that discipline by default, and adding version de-dup the others lack. It isn't magic per-token compression.

Takeaway

If you're answering questions over a corpus with real document versioning, the tokens you waste aren't in chunk size - they're in sending the LLM the same document five times and in synthesis modes that call the model once per chunk. SWIRL 5's design removes both by default. Measure your own stack the same way: total input and output tokens per query, summed across every LLM call. That number, not top_k, is your bill.

Comments

[Vinicius Pereira]

The versioned-corpus scenario is worse than the token math shows, because the damage starts one step before the bill. Top_k is a fixed aperture allocated by similarity score, and near-identical versions embed to near-identical vectors, so the copies cluster at the top of the ranking. In your 5-versions-plus-3-documents scenario, LangChain's default retriever (similarity search, k=4) can hand the model nothing but copies of the same policy. Every slot a copy occupies is one unique document displaced, and copies 2 through 5 carry close to zero marginal information. Effective recall over distinct evidence collapses while every similarity score looks excellent, and no token budget, however generous, brings back a document the retriever never returned. MMR exists for exactly this failure and ships as an opt-in search_type, not the default, which lines up with your "none of them dedup by default" finding.

And when the versions actually disagree, which is the only reason versions exist, it compounds: which policy figure reaches the model is decided by embedding-space accident, not by recency or canonicity. If the superseded figure arrives in four copies and the current one in zero, the model reads repetition as corroboration and answers with the stale majority, fluently and at full price. Dedup is a correctness mechanism that happens to save tokens.

One addition to your takeaway, which already says to measure output too: the chart prices the input side, but refine mode also regenerates the full draft of the answer at every step. Only the final draft is delivered, yet all N drafts are billed at output rates, several times the input rate on every major provider, and each discarded draft is re-billed as input to the next call. So the multi-call bars are floors. The per-query ledger you describe, both directions, summed across every call, is the only accounting that catches that.