Semantic Chunking: Split Documents at the Seams of Meaning to Fix RAG Retrieval

Retrieval-augmented generation never feeds the whole document to the model — it feeds chunks. So the answer the model can give is bounded by what a single chunk contains. If chunking splits an idea in half, the top-retrieved chunk is half an answer, and no amount of clever prompting recovers the missing half. Chunking is an upstream quality decision, made before a single query runs — and it quietly decides how good your RAG system is.

The naive way cuts mid-thought

Fixed-size chunking cuts every N characters (or sentences) regardless of meaning. It's trivial and needs no embeddings, but the boundary falls wherever the counter runs out — often straight through a topic. A definition and its example land in different chunks; a question and its answer get separated. The information is all there, just never together in one retrievable unit.

Split where the meaning changes

Semantic chunking splits at the seams of meaning instead. Embed each sentence, measure how similar each one is to its neighbour, and drop a breakpoint wherever adjacent similarity falls. Same topic continuing → high similarity; a topic change → the vectors point elsewhere and cosine drops.

The demo embeds with a lightweight TF-IDF bag-of-words (deterministic, no API), then walks the sentences measuring adjacent cosine — this sequence of n−1 numbers is the entire signal the chunker splits on:

def adjacent_sims(vecs):
    return [cosine(vecs[i], vecs[i + 1]) for i in range(len(vecs) - 1)]

sims = adjacent_sims(vecs)
# e.g. [0.55, 0.41, 0.60, 0.00, 0.48, ...] -> the 0.00 is a topic change

The breakpoint-percentile threshold

Where is a drop "big enough" to cut? A fixed cutoff like sim < 0.3 breaks on documents whose similarities live in a different range. Instead, work in distance = 1 − similarity and set the threshold at a percentile of the gaps. p = 85 splits only the top 15% biggest jumps; a lower percentile lowers the bar, so more gaps qualify and you get more, smaller chunks. This is LangChain's breakpoint_threshold_type="percentile":

def percentile(xs, p):
    s = sorted(xs)
    if not s: return 0.0
    i = (p / 100) * (len(s) - 1)
    lo, hi = math.floor(i), math.ceil(i)
    return s[lo] + (s[hi] - s[lo]) * (i - lo)   # linear interpolation

def breakpoints(sims, p=85):
    dist = [1 - s for s in sims]
    thr  = percentile(dist, p)
    return {i for i, d in enumerate(dist) if d >= thr}   # gaps to cut after

# lower p -> lower thr -> more gaps clear it -> more, smaller chunks

Assemble the chunks

Walk the sentences; a breakpoint after sentence i starts a new chunk at i+1. Everything between two breakpoints is a run of mutually-similar sentences — one coherent, self-contained idea:

def semantic_chunks(sents, p=85):
    vecs  = tfidf_vectors(sents)
    sims  = adjacent_sims(vecs)
    brks  = breakpoints(sims, p)
    chunks, cur = [], [sents[0]]
    for i in range(len(sents) - 1):
        if i in brks:                 # seam between sentence i and i+1
            chunks.append(" ".join(cur)); cur = []
        cur.append(sents[i + 1])
    chunks.append(" ".join(cur))
    return chunks

On a document that runs photosynthesis → Roman empire → espresso, a good percentile lands the two breakpoints exactly on the real topic seams, so each chunk is a single clean topic — and the retriever returns a whole answer, not half a story.

The dials and the toolkit

Chunks too big dilute the embedding and stuff the prompt with irrelevance; too small and a fact loses the context that makes it answerable. A little overlap — repeat the last sentence or two into the next chunk — insures against a breakpoint that split a claim from its evidence. Semantic chunking sets where to cut; size and overlap tune how much context rides along.

In production, swap the TF-IDF toy for a real embedder (Anthropic has no embeddings endpoint — it recommends Voyage AI, or use LangChain's SemanticChunker), index the chunks, retrieve the top few, and hand them to the model. Recursive chunking respects structure; semantic chunking respects meaning — reach for it when documents mix topics with no clean structure and retrieval quality matters.

Slide the percentile, watch the sentence strip re-colour into semantic chunks, and compare retrieval side-by-side against fixed-size here: https://dev48v.infy.uk/ai/days/day55-semantic-chunking.html