Chunked a Document and Found a New Retrieval Problem

Chunking means splitting a long document into smaller overlapping pieces and embedding each one separately, instead of embedding the whole thing or lopping off the end like Entry 05's truncation hack. It fixes the content-loss problem cleanly — nothing gets silently dropped. What it doesn't fix, and what I didn't see coming until I was staring at the results, is a structural bias: a document split into 17 pieces now has 17 separate shots at showing up in search results, while a short document still only gets one. More chunks means more chances to rank, whether or not any individual chunk is actually the most relevant thing in the store.

Chunked a new draft (17 pieces, 1000 characters each with 200-character overlap) and went to embed it into the same collection from the last two entries. Hit two real snags getting there.

First one: reconnecting to what I thought was my collection returned something almost empty — just the new chunks, none of the three entries from before. Turns out PersistentClient(path="./chroma_db") is relative to wherever you launch Python from, and I'd started this session in a different folder than the earlier ones. Went digging with find across the filesystem and turned up three separate chroma_db folders sitting around. The "empty" one wasn't data loss — it was a brand new database I'd created by accident just by being in the wrong directory. Fixed by cd-ing back to the right place before reconnecting.

Second snag, after that was sorted: an early query using collection.query(query_texts=[...]) blew up with InvalidArgumentError: Collection expecting embedding with dimension of 768, got 384. Passing raw text instead of a pre-computed embedding makes Chroma quietly fall back to its own default embedding model, which spits out a different vector size than nomic-embed-text. Same lesson as last entry, just a different way to hit it: always embed the query with the same model you used for the documents.

With both of those out of the way, actually chunked and loaded the thing:

import ollama

text = open("managed-vs-self-hosted-handing-over-keys.md").read()

chunk_size = 1000
overlap = 200
chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)]

for i, chunk in enumerate(chunks):
    resp = ollama.embeddings(model="nomic-embed-text", prompt=chunk)
    collection.upsert(
        ids=[f"managed-vs-self-hosted-handing-over-keys_chunk{i}"],
        embeddings=[resp["embedding"]],
        documents=[chunk],
    )

collection.count()   # → 20 (3 original entries + 17 new chunks)

Then ran three queries — the same oc/pizza pair from last time, plus a real question about what the new document actually covers:

q1_embed = ollama.embeddings(model="nomic-embed-text", prompt="how do I check pod status with oc")
q1 = collection.query(query_embeddings=[q1_embed["embedding"]], n_results=3)

q2_embed = ollama.embeddings(model="nomic-embed-text", prompt="what's the best pizza topping")
q2 = collection.query(query_embeddings=[q2_embed["embedding"]], n_results=3)

q3_embed = ollama.embeddings(model="nomic-embed-text", prompt="What are my options for kubernetes, should I use managed or self-hosted Kubernetes")
q3 = collection.query(query_embeddings=[q3_embed["embedding"]], n_results=3)

print("Query 1:", q1["ids"], q1["distances"])
print("Query 2:", q2["ids"], q2["distances"])
print("Query 3:", q3["ids"], q3["distances"])

Query	Top 3 matches	Distances
"how do I check pod status with oc"	correct file, then 2 unrelated chunks	437.72, 450.32, 453.22
"what's the best pizza topping"	3 unrelated chunks, all from the new doc	519.80, 531.01, 531.51
"managed or self-hosted Kubernetes"	3 correct chunks from the new doc	290.34, 314.06, 316.37

The Kubernetes question is the cleanest result the series has produced — every one of the top 3 came from the right document, at distances meaningfully tighter than anything I'd seen before. Chunking clearly works for making a long document's content actually findable.

But look at the oc question. Last entry, its #2 result was the genuinely-related URL entry, distance 499.63. Here, that document got shoved entirely out of the top 3 — replaced by two chunks that have nothing to do with the question, just because the 17-chunk document has more entries competing for the middle of the ranking. Not more relevant. More numerous.

So: chunking really did fix the content-loss problem, and the on-topic result here is genuinely the best this series has produced. But it's not a free upgrade. A document with a lot of chunks crowds out equally-relevant documents that only got embedded once, purely by having more shots at ranking. The usual fix for this in production RAG is a per-document result cap or a re-ranking pass after retrieval — that's the obvious next thing to try, rather than just assuming more chunks is always better search.