LangChain vs LlamaIndex vs Chonkie: same 94-page PDF

LlamaIndex. Its SentenceSplitter gave me the best recall@5 (0.86, against 0.79 for LangChain's recursive splitter) on a 94-page policy PDF, without writing a custom separator list. LangChain wins if you're already deep in LCEL. Chonkie is the fastest by a wide margin and the one I'd pick for a batch job over a million documents. Chunk size mattered more than the library did.

Quick disclosure before anything else: the RAG chunk size calculator I link to below is one I built. I got tired of re-deriving the same token math in a scratch file, and the four existing pages I found all assumed OpenAI's tokenizer and 1,000-character chunks. Mine is free, runs entirely in your browser, no signup, nothing uploaded. If you know a better one, tell me and I'll link it instead.

The task: one 94-page policy PDF and 38 questions

Last Tuesday I got handed a commercial property insurance policy and a support inbox. The ask was ordinary: answer questions like "what's the deductible for wind damage during a named storm" without a human reading 94 pages every time.

I pulled the text with pdftotext -layout, which gave me 214,883 characters. Then I sat down and wrote 38 questions by hand, each paired with a "needle" string that appears on exactly one page. That labelling took 47 minutes and it's the only reason any number below means anything. A chunking benchmark without labels is just vibes with decimal places.

Same setup for every run: BAAI/bge-small-en-v1.5 as the embedding model, normalized vectors, cosine similarity, top 5 results. The metric is recall@5, the fraction of questions where at least one of the top 5 chunks contains the needle.

Here's the scoring script. No framework, no vector database, just numpy:

# eval_chunks.py - score a chunker by recall@5 on hand-labelled questions
import json, time
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

def recall_at_k(chunks, questions, k=5):
    cv = model.encode(chunks, normalize_embeddings=True, batch_size=64)
    qv = model.encode([q["text"] for q in questions], normalize_embeddings=True)
    hits = 0
    for q, v in zip(questions, qv):
        top = np.argsort(-(cv @ v))[:k]
        if any(q["needle"].lower() in chunks[i].lower() for i in top):
            hits += 1
    return hits / len(questions)

def score(name, split_fn, text, questions):
    t0 = time.perf_counter()
    chunks = split_fn(text)
    dt = time.perf_counter() - t0
    r = recall_at_k(chunks, questions)
    print(f"{name:11} chunks={len(chunks):4}  split={dt:6.2f}s  recall@5={r:.2f}")

if __name__ == "__main__":
    text = open("policy.txt", encoding="utf-8").read()
    questions = json.load(open("questions.json", encoding="utf-8"))

    from langchain_text_splitters import RecursiveCharacterTextSplitter
    lc = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
        chunk_size=512, chunk_overlap=64)
    score("langchain", lc.split_text, text, questions)

    from llama_index.core.node_parser import SentenceSplitter
    li = SentenceSplitter(chunk_size=512, chunk_overlap=64)
    score("llamaindex", li.split_text, text, questions)

    from chonkie import RecursiveChunker
    ch = RecursiveChunker(chunk_size=512)
    score("chonkie", lambda t: [c.text for c in ch(t)], text, questions)

Three lines of output. Then you get to argue with them.

LangChain: I was wrong about this one for an hour

First run, LangChain came dead last. 0.44 recall against LlamaIndex's 0.86. That gap felt too big to be real, and it wasn't.

RecursiveCharacterTextSplitter(chunk_size=512) counts characters. SentenceSplitter(chunk_size=512) counts tokens. Same parameter name, same value, and for English prose that's roughly a 4x difference in how much text lands in each chunk. My careful apples-to-apples comparison was quietly pitting 512-character chunks against chunks of about 2,000 characters. I'd made this exact mistake in April on a different project and still walked straight into it again.

Switching to RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=512, chunk_overlap=64) fixed it: 137 chunks, 0.79 recall, 2.8 seconds. Perfectly respectable.

The thing that still bugs me is what that helper counts with. It pulls in tiktoken and defaults to cl100k_base, an OpenAI tokenizer. I'm embedding with a BERT-family model that uses WordPiece. On plain English the two agree within about 8%, so nothing explodes, but on code, JSON blobs or non-Latin scripts the drift gets ugly and your "512-token" chunks start overflowing a 512-token encoder. Silent truncation, no warning.

Import paths are also a mess if you're following an older tutorial. It's langchain_text_splitters now, and half the search results still say from langchain.text_splitter import.

LlamaIndex: the boring one that won

SentenceSplitter at 512 tokens with 64 overlap: 129 chunks, 0.86 recall, 4.1 seconds. It was the slowest of the three on a single document because it actually runs a sentence tokenizer before doing anything else.

That sentence tokenizer is the whole reason it won. Insurance definitions read like "Named Storm means any storm or weather disturbance that is named by the National Weather Service." Cut that in the middle and neither half retrieves for a question about named storms. The recursive splitters get this right most of the time via their separator list, but "most of the time" across 129 chunks is a handful of dead ones.

Two things I didn't love. The install is heavy: llama-index-core was 41 MB in a fresh venv against 2.9 MB for chonkie. If chunking is the only thing you want, that's a lot of framework to carry.

Bigger issue is the defaults. SentenceSplitter() with no arguments is chunk_size=1024, chunk_overlap=200. On my document that scored 0.68. Nobody's default is tuned for your corpus, and this one is off by enough to matter.

Chonkie: fast, small, and it surprised me

Chonkie is the small library in this comparison and I expected it to lose. It didn't. RecursiveChunker at 512 tokens gave 133 chunks and 0.82 recall in 0.38 seconds. That's within noise of LlamaIndex on quality and roughly 10x faster.

Speed stops being an academic concern once the corpus grows. I ran all three over the client's full document set (1,240 files, 2.1 GB of extracted text). Chonkie finished in 47 seconds. LangChain took 3 minutes 4 seconds. LlamaIndex took 6 minutes 12 seconds. For a one-time ingest, who cares. For a nightly re-index that has to finish before the morning, that's the difference between a cron job you forget about and a Slack alert at 2am.

I also tried SemanticChunker, which embeds each sentence and splits where similarity drops. 0.84 recall, 11.3 seconds on one document. Thirty times the cost for two points I can't distinguish from measurement noise on 38 questions. On a genuinely mixed corpus it might earn its keep. Here it didn't.

The rough edge is polish. The API moved between the version most blog posts describe and the 0.5.x I installed, so half the snippets I found were wrong. And when I passed a tokenizer name it didn't recognise, I got a bare KeyError out of a dict lookup with no hint about what the valid names are. That cost me twenty minutes I'd rather have spent elsewhere.

The scores, and which one I'd actually ship

Criterion	LangChain	LlamaIndex	Chonkie
Best recall@5, 38 questions	0.79	0.86	0.82
What chunk_size counts by default	characters	tokens	tokens
Chunk one 94-page doc	2.8s	4.1s	0.38s
Chunk 1,240 docs	3m 04s	6m 12s	47s
Install size, fresh venv	8.4 MB	41 MB	2.9 MB
Error on a bad tokenizer name	named ValueError	named ValueError	bare KeyError
Found the answer in docs under 2 min	yes	yes	no
Defaults I'd ship unchanged	no (characters)	no (1024/200)	yes

I'd use LlamaIndex's SentenceSplitter for anything under roughly ten thousand documents where answer quality is the point. Above that, or anywhere re-indexing runs on a schedule, Chonkie. If your pipeline already lives in LangChain, stay there and just call from_tiktoken_encoder explicitly, because the character default will bite you and it won't be loud about it.

Here's the finding I keep coming back to, though. Across the three libraries at their best settings the spread was 0.07. Across chunk sizes with a single library, it was 0.18: 256 tokens scored 0.74, 512 scored 0.86, 768 scored 0.81, 1024 scored 0.68. The parameter beat the vendor by more than double. If you're agonising over which splitter to import before you've swept chunk size, you're optimising the wrong variable.

That gap is why I built the RAG chunk size calculator. Feed it your embedding model and the kind of document you're indexing, and it hands back a starting chunk size and overlap along with the token math (context window, characters per token for your tokenizer, how many chunks that means for a document of size N). It's a starting point, not an oracle. You still need your own 30 labelled questions.

FAQ

Q: Does overlap actually help, or is it cargo cult?

A: It helps, less than people assume. Zero overlap scored 0.79, 64 tokens scored 0.86, 128 tokens scored 0.87 while producing 22% more chunks to store and search. I settled on 64, about 12.5% of chunk size, and that ratio has held up on two other projects since.

Q: Why not just use semantic chunking everywhere?

A: Because on this document it bought 0.02 recall for 30x the processing time. Semantic chunking pays off when a single file jumps between unrelated topics. A structured policy document already has that structure in its headings.

Q: Will these numbers hold for my documents?

A: Almost certainly not, and I'd be suspicious of anyone who told you otherwise. Legal and insurance prose is dense, repetitive and full of defined terms, which flatters sentence-aware splitting. Chat logs or source code behave differently. Copy the script above, label 30 questions of your own, rerun it. Mine took 47 minutes.

Q: What about markdown and code files?

A: Different tools. Use a header-aware splitter for markdown so you keep section context attached, and Chonkie's CodeChunker (or a tree-sitter based splitter) for source, so functions stay whole. Splitting code on blank lines destroys exactly the boundaries you want to retrieve on.

Written with AI assistance and human review. Try the tool at aidevhub.io/rag-chunk-calculator.