Measuring whether semantic chunking actually helps, on one fixed corpus

Everyone repeats that semantic chunking beats fixed-size chunking for RAG. I wanted to know if that was true for my corpus, or just a thing people say. So I built the comparison into EvalBench: one corpus, one query set, three chunking strategies, and a single precision number I could read off a leaderboard.

The result I care about here is not a benchmark score. It is the setup. Getting a fair comparison between chunkers is harder than writing the chunkers, because almost everything that makes retrieval look good or bad has nothing to do with how you split the text.

The trap: comparing chunkers by comparing pipelines

If you swap the chunker and also swap the embedder, or the corpus, or the top-k, you have not measured the chunker. You have measured a different pipeline. Most "semantic wins" posts do exactly this, and you cannot tell how much of the win is the split versus everything else that moved with it.

So in EvalBench the only thing that varies is the split. A RAG run is identified by an embedder::chunk_strategy string, parsed apart before anything runs:

def parse_pipeline_model(model: str) -> tuple[str, ChunkStrategy]:
    """Parse the exact ``embedder::chunk_strategy`` row identifier."""
    ...
    if strategy not in _ALLOWED_STRATEGIES:
        raise ValueError(f"unknown chunk strategy {strategy!r}")

Same embedder, same 200-document corpus, same 15 queries across finance, legal, medical, physics, and software. The strategy is the one knob. Everything downstream is shared code.

The three strategies

There are three, not two. The interesting comparison is not fixed versus semantic, it is fixed versus recursive versus semantic, because recursive is what most people actually ship.

fixed_512 is the dumb one. Whitespace-tokenize the whole document, cut it into 512-token windows with a 64-token overlap, done. It respects nothing about the text.

def _window_token_groups(tokens, window=512, overlap=64):
    groups, start = [], 0
    while start < len(tokens):
        end = min(start + window, len(tokens))
        groups.append(list(tokens[start:end]))
        if end == len(tokens):
            break
        start = end - overlap
    return groups

recursive splits on structure first. Try paragraph breaks, then single newlines, then sentence boundaries, and only fall back to a hard token cut when a piece is still too big. Then it repacks those units into bounded chunks. This is the LangChain-style splitter, and it is a real baseline, not a strawman.

semantic is the one people get excited about. Embed every sentence, walk them in order, and start a new chunk when the meaning drifts. Drift is measured as cosine similarity between adjacent sentences dropping below a threshold:

low_similarity = (
    len(current_sentences) >= 3
    and cosine_similarity(vectors[index - 1], vectors[index]) < 0.65
)
too_large = len(current_tokens) + len(sentence_tokens) > 512
if current_tokens and (too_large or low_similarity):
    flush()

Two things in that snippet matter and neither is obvious from the marketing. The >= 3 guard means a chunk needs at least three sentences before a similarity drop is allowed to break it, so one weird sentence early on does not shatter a chunk into fragments. And the too_large check means semantic still has a hard 512-token ceiling. So it is not really "split on meaning." It splits on meaning until it hits the same 512-token window fixed_512 uses, then cuts. That ceiling is what keeps the comparison fair. If semantic could emit arbitrarily long chunks, it would win on precision for a boring reason: bigger chunks catch more gold documents.

The metric has to be simple enough to trust

I score every strategy on context precision. Take the top 10 ranked chunks, count how many come from a document in the gold set, divide:

retrieved_chunks = ranked_chunks[:10]
context_precision = (
    sum(chunk.doc_id in gold_doc_ids for chunk in retrieved_chunks)
    / len(retrieved_chunks)
    if retrieved_chunks
    else 0.0
)

That is deliberately crude. It counts a chunk as relevant if its source document is relevant, not if the chunk text itself answers the query. A stricter metric would grade chunk content, but then I would be measuring my grader as much as my chunker, and I already have a separate faithfulness judge for content quality. For a chunker comparison, "did we pull chunks from the right documents" is the question, and document-level precision answers it without dragging an LLM into the measurement.

Every metric is clamped to [0, 1] with a hard failure if it lands outside, so a broken chunker that emits duplicate or malformed chunks throws instead of quietly publishing a nonsense score:

def _clamp_metric(value):
    tolerance = 1e-12
    if not math.isfinite(value) or value < -tolerance or value > 1.0 + tolerance:
        raise ValueError(...)
    return float(min(1.0, max(0.0, value)))

What broke, and what I would do differently

The gold labels are per document, not per chunk. That is the real limitation here. If a relevant document is 40 chunks long and only 2 of those chunks actually answer the query, my precision metric happily credits all 40 as relevant when they surface. So this setup answers "does this strategy retrieve from the right documents," and it is blind to whether it retrieves the right passage inside them. For a first pass on a 200-document corpus that is a reasonable trade, but it means a chunker that fragments a relevant document into many pieces can look better than one that pulls a single tight passage, purely because more of its chunks share a gold doc_id. Chunk-level labels would fix this and are the obvious next step.

The other thing I would change is the sample size before trusting any ranking. 15 queries, 3 per domain, is enough to see a large effect and nowhere near enough to call a small one. EvalBench already publishes Wilson intervals next to every mean for exactly this reason, so the leaderboard shows the uncertainty instead of hiding it. If two strategies land within each other's intervals at n=15, the correct read is "no measured difference," not "semantic wins by a hair." I would rather report a wide interval than a confident wrong number.

Takeaway

If you want to know whether semantic chunking helps, hold the embedder, corpus, queries, and token ceiling fixed, and vary only the split. Then read the precision with its confidence interval attached. A comparison where three things moved at once tells you nothing about the one you changed.