A Cross-Encoder Reads the Query and the Document Together, Which Your Embedding Model Structurally Cannot

Level 5 of nine in Project Arc Rector - an agentic RAG stack built from free, self-hostable parts, one swappable level at a time. Level 4 was the vector database. This is the layer above it with more leverage, because an index only returns what is near, and the embedding decides what near means.

The page, with a real hashing embedder, a real distributional one and a real pair scorer computing in your browser: https://dev48.infy.uk/arcrector/level5-embeddings.html
Repo: https://github.com/dev48v/arc-rector

The layer that fails quietly

A misconfigured vector database throws. A misconfigured embedding model returns 768 perfectly well-formed floats and then retrieves slightly worse documents forever.

The cheap open models are competitive and all self-hostable: Nomic Embed (137M parameters, 768 dimensions, Apache-2.0, about 274 MB, served over plain HTTP by Ollama), Jina v2 small at 512 dimensions, MiniLM at 384. Two of those numbers are the trap - read dim off the checkpoint, because a collection created at 768 accepts nothing from a 384-dim model.

The other trap is asymmetry. Nomic is instruction-prefixed, search_document: on stored chunks and search_query: on questions, and omitting them raises nothing:

def embed_documents(self, texts):
    return self._embed([DOCUMENT_PREFIX + t for t in texts])
def embed_query(self, text):
    return self._embed([QUERY_PREFIX + text])[0]

On the page's own harness, getting them wrong moves 6 of 7 result sets and drops mean recall@4 from 0.738 to 0.667 omitted and 0.619 mismatched - while nDCG drifts slightly up, which is reported rather than hidden. Symmetric models take no prefix, and copying Nomic's onto them is its own quiet bug.

Dimensionality is a bill, not a quality knob

384 dimensions at float32 is a quarter of the bytes of 1024, and Matryoshka models (Nomic v1.5, OpenAI's v3 family) are trained so a prefix of the vector stays useful, which lets one model serve several budgets. Truncating an ordinary embedder is a different operation and can be much worse.

The real win is usually the reranker

A bi-encoder pins every document in space before your question exists. A cross-encoder reads the query and one document together in a single pass, so it can attend across them - which a bi-encoder structurally cannot, and which is why nothing about it can be precomputed. It carries no dim and has nothing to store.

vector = deps.embeddings.embed_query(question)          # search_query: prefix
hits   = deps.store.search(vector, top_k=deps.fetch_k)  # 12 cheap candidates
return deps.reranker.rerank(question, hits, deps.top_k) # 4 accurate ones

Retrieve wide with the cheap model, rerank narrow with the expensive one. The gap between fetch_k and top_k is the only room the reranker has; set them equal and it becomes a sort of the four items that were already the four. And replace the vector score, never average it - a cosine and a cross-encoder logit are different scales, and blending them is the commonest way to make reranking worse than none.

What this level does not fix

Hybrid retrieval is the lever it does not pull. BM25 catches the exact rare token - a product code, an error string - that a dense vector smooths away, and fusing two ranked lists with reciprocal rank fusion needs no score calibration, which is why RRF beats a weighted sum of incomparable scores.

Nor does any of this rescue bad chunks, which is Level 6: ingestion and parsing, where the chunks come from. And you cannot tell whether it helped without a labelled query set of your own - the gold set on the page is seven queries graded 0 to 3, and says of itself that this is a smoke test rather than an evaluation. Fifty real questions with known answers beats every published benchmark for your corpus.

The whole stack, nine levels, all free to self-host: https://dev48.infy.uk/arcrector.php