The answer was in the chunk. I could see it. Paragraph four of a 600-token chunk about our refund policy, word for word what the user asked. And my RAG pipeline refused to retrieve it, no matter how I rephrased the query.
The bug wasn't in the vector database, the prompt, or the LLM. It was one number buried in a model config: all-MiniLM-L6-v2 truncates at 256 tokens, silently. Everything after token 256 of my chunk never made it into the embedding. Paragraph four didn't exist as far as retrieval was concerned.
If you built a RAG pipeline from a tutorial in the last few years, there's a decent chance you have the same bug right now.
TL;DR
-
sentence-transformers/all-MiniLM-L6-v2hasmax_seq_length = 256. Input longer than that is cut off with no error and no warning. - The embedding only represents the first ~256 WordPiece tokens of each chunk. Text past that point is stored in your DB but can never be matched by a query.
- Your chunker probably counts characters or tiktoken tokens, not the embedding model's tokens, so you don't notice the overflow.
- Fix: chunk with the embedding model's own tokenizer and keep chunks under its limit, or switch to a long-context embedding model.
- Detection is a 10-line script. Run it before you tune anything else.
Why does all-MiniLM-L6-v2 truncate at 256 tokens?
all-MiniLM-L6-v2 truncates because sentence-transformers sets max_seq_length = 256 for it, and the tokenizer is called with truncation=True. Any token past position 256 is dropped before the model ever runs.
The model card says this out loud if you read it: input longer than 256 word pieces is truncated. It was also trained on sequences of only 128 tokens, so even the 256 limit is a stretch from what it learned on.
You can check in two lines:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
print(model.max_seq_length) # 256
The underlying BERT architecture has 512 position embeddings, so the hard ceiling is 512. But the default is 256, and almost nobody changes it.
Here's the part that bites: model.encode() happily accepts a 5,000-token string and returns a perfectly normal 384-dimensional vector. Same shape, same norm, same vibe. It just describes the opening of your text and nothing else.
How does embedding truncation break RAG retrieval?
Truncation breaks retrieval because the vector stored for a chunk only encodes its first 256 tokens, while the full chunk text sits next to it in the database. The LLM would see the whole chunk if retrieved. The retriever only ever sees the head.
So you get a very specific failure pattern:
- Questions answered in the first paragraph of a chunk retrieve fine.
- Questions answered deeper in the chunk retrieve badly or not at all.
- You open the chunk in your DB, see the answer sitting right there, and lose an afternoon blaming the query.
My setup was a textbook case. I used a splitter configured for 512 tokens, measured with tiktoken. Then I embedded with MiniLM. Two problems stacked:
- 512 is already double the 256 limit.
- tiktoken's
cl100k_baseand BERT's WordPiece don't count the same way. WordPiece has a 30k vocabulary and splits rare words, identifiers and code into more pieces. A "512-token" chunk of API docs can easily be 600+ WordPiece tokens.
Rough math on my own chunks: if a chunk is ~600 WordPiece tokens and the model sees 254 of them (two slots go to [CLS] and [SEP]), then around 58% of every chunk was invisible to search. Not noisy. Invisible.
Why don't you notice it in testing?
You don't notice because most test questions are written by looking at the start of a document. When I write "eval questions" by skimming, I skim the top. Headings, intros and first paragraphs are exactly what the truncated embedding captures, so the eval looks great.
The other reason: retrieval still returns something. Top-k always has k results. A semantically nearby chunk whose first paragraph mentions "refunds" will outrank the chunk that actually answers "can I get a refund after 30 days?" if that answer lives on line 20. The LLM then produces a confident, slightly wrong answer from the neighbor chunk. Nothing crashes. It just gets worse in a way that looks like a model quality problem.
How do I check if my embeddings are being truncated?
Tokenize every chunk with the embedding model's own tokenizer and count how many exceed max_seq_length. That's the whole check:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
tok, limit = model.tokenizer, model.max_seq_length
def audit(chunks):
lengths = [len(tok(c)["input_ids"]) for c in chunks] # includes [CLS]/[SEP]
over = [n for n in lengths if n > limit]
lost = sum(n - limit for n in over)
total = sum(lengths)
print(f"{len(over)}/{len(chunks)} chunks truncated")
print(f"{lost / total:.1%} of all tokens never embedded")
print(f"longest chunk: {max(lengths)} tokens (limit {limit})")
audit(my_chunks)
Run this against a sample of what's actually in your vector store, not against your splitter settings. The settings tell you what you intended. The audit tells you what happened.
If the second line prints anything above a couple of percent, you've found a retrieval bug that no amount of prompt engineering will fix.
How do I fix embedding model truncation in a RAG pipeline?
Make the chunk size and the embedding model's token limit agree, measured in the same tokenizer. There are four ways to get there, roughly in order of effort.
1. Chunk with the embedding model's tokenizer
This is the real fix. LangChain ships a splitter for exactly this:
from langchain_text_splitters import SentenceTransformersTokenTextSplitter
splitter = SentenceTransformersTokenTextSplitter(
model_name="sentence-transformers/all-MiniLM-L6-v2",
tokens_per_chunk=200, # headroom under 256
chunk_overlap=30,
)
chunks = splitter.split_text(document)
If you'd rather keep your existing recursive splitter, pass it a length function built on the model's tokenizer instead of len or tiktoken. The point is the same: count tokens the way the embedder counts them.
Leave headroom. I target ~80% of the limit so a chunk with a weird code block doesn't tip over.
2. Add an assertion at ingest time
Truncation is silent, so make it loud:
n = len(tok(chunk)["input_ids"])
if n > limit:
raise ValueError(f"chunk {chunk_id} is {n} tokens, embedder limit is {limit}")
This one line would have saved me a day. It also catches the day someone swaps the embedding model and forgets the limit changed.
3. Don't just raise max_seq_length
You can set model.max_seq_length = 512 and the model will run. It won't crash, because BERT has 512 position slots. But MiniLM was trained on 128-token inputs. Pushing it to 512 means it's embedding sequences 4x longer than anything it saw in training, and mean pooling over 512 tokens smears the signal. It's a valid experiment, not a free fix. Measure recall on your own eval set before and after.
Going past 512 fails outright, because there are no position embeddings for those slots.
4. Use a long-context embedding model
If your documents are naturally long and chunking them small destroys context, pick a model built for it. BAAI/bge-m3 and nomic-embed-text-v1.5 both support 8192 tokens. Hosted APIs like OpenAI's text-embedding-3-small accept up to 8191 tokens and return an error if you go over, which is honestly the behavior I wanted all along.
But remember: a longer window isn't automatically better retrieval. One vector summarizing 3,000 tokens is a blurry average of many topics. Small, focused chunks often still win. The goal is chunks that fit, not chunks that are big.
Which embedding models truncate silently?
Most local sentence-transformers models truncate silently at their max_seq_length, because truncation is the library's default behavior. The limit varies per model, so never assume:
| Model | Default max_seq_length |
|---|---|
| all-MiniLM-L6-v2 | 256 |
| all-mpnet-base-v2 | 384 |
| BAAI/bge-small-en-v1.5 | 512 |
| BAAI/bge-m3 | 8192 |
Check model.max_seq_length for whatever you load. Cross-encoder rerankers do the same thing on the query + passage pair, so a reranker with a 512 limit and a long query also sees less of the passage than you think.
The checklist I use now
- Chunk size is defined in the embedding model's tokens, not characters or tiktoken.
- Chunks target ~80% of
max_seq_length. - Ingest raises on overflow instead of truncating.
- Eval questions are sampled from the middle and end of chunks, not just the top.
- Swapping embedding models triggers a re-audit.
None of this is exotic. It's just the one config value that every quickstart skips.
So, does all-MiniLM-L6-v2 really truncate at 256 tokens?
Yes. all-MiniLM-L6-v2 truncates input at 256 WordPiece tokens by default in sentence-transformers, and it does it silently: encode() returns a normal-looking vector that only represents the start of your text. In a RAG pipeline, that means any fact past roughly the first 250 tokens of a chunk can never be retrieved, even though it's stored in your database. Audit your chunk lengths with the model's own tokenizer, split with SentenceTransformersTokenTextSplitter or an equivalent token-aware splitter under the limit, and add an ingest-time assertion so truncation fails loudly instead of quietly making your retrieval worse.
Written by the developer behind Preterview, an interview prep platform.
Top comments (0)