Understand vectors, similarity and semantic search through a small worked example, then learn what similarity cannot tell you.
Originally published on Stack & Craft.
What you will learn: Explain embeddings and cosine similarity, compare semantic and keyword search, and inspect whether a retrieved passage actually answers the question.
Prerequisites: Start with the Building and Evaluating a RAG System guide. Worked exercises use Python’s standard library and need no paid model.
The problem: different words, related ideas
An employee asks about vacation. The handbook calls it annual leave. Exact-word search may miss the connection. Semantic search uses a learned representation of text to compare related meanings, although it can also retrieve a related passage that does not answer the question.
Embedding
An embedding is a list of numbers produced by a model to represent an input. Text embeddings encode useful patterns learned from data. The dimensions of a real model are not generally interpretable labels such as “vacation” or “finance”.
Different texts can have nearby representations without containing identical words.
Vector
A vector is an ordered list of numbers. Comparing vector representations helps a search system rank candidates. Real text embeddings usually have many dimensions; the two-dimensional sketch in this guide is only a teaching illustration.
[1, 0] and [0.8, 0.6] are two toy vectors, not actual model outputs.
Semantic similarity
A measure of how close representations are under the chosen comparison rule. Similarity can reflect topic or phrasing, but it does not prove factual correctness, authority, access rights or answerability.
A refund-request form may be similar to a refund-deadline question while omitting the deadline.
Work through cosine similarity
Cosine similarity compares the direction of two vectors. It is the dot product divided by the product of their lengths. A dot product multiplies corresponding numbers and sums the results. A vector length is the square root of the sum of its squared numbers.
A calculation you can reproduce
Let the query be q=[1,0]. Candidate A=[0.8,0.6] has length 1, so its cosine score is 0.8. Candidate B=[0,1] also has length 1, but its score is 0. A ranks ahead of B under this metric. These numbers were chosen to teach the calculation; they do not measure real text meaning. A zero-length vector needs explicit handling because division by zero is undefined.
Follow the two paths
Use compatible query and document representations. If you change an embedding model, dimensions or its preprocessing, plan re-embedding and validate compatibility. Reusing old document vectors with an unrelated query model can make ranking meaningless.
Prepare first; search at question time
- Prepare passages — Read and chunk documents. Keep the original text, source ID, version and permissions.
- Embed and index — Encode each passage with the chosen document encoder and store its vector with its text and metadata.
- Encode the question — Use the model’s supported query encoder or query mode, including any required prefix.
- Filter and rank — Enforce authorization before results reach the model. Rank candidates using the index’s supported similarity measure.
- Inspect evidence — Read the returned passage. Verify that it contains the needed fact before using it in an answer.
Choose search signals for the question
Keyword and semantic retrieval solve different kinds of mismatch. Hybrid search combines signals, but combining raw scores carelessly can let one scale dominate the other. Evaluate a documented fusion method on representative questions.
| Situation | What it means | Next check |
|---|---|---|
| Exact error code AUTH-403 | Characters are important; semantic similarity may retrieve related but different codes. | Establish a keyword baseline and inspect exact matches. |
| Vacation versus annual leave | A wording mismatch may benefit from semantic retrieval. | Check whether the matched passage contains the requested rule. |
| A policy restricted to HR | Similarity says nothing about permission. | Filter by the current user’s access before returning the passage. |
| A multilingual question | Language coverage depends on the selected model and collection. | Evaluate real language pairs; do not assume translation quality. |
Do the exercise
Run exercises/embeddings_demo.py in the downloadable RAG project. It uses only Python’s standard library and clearly labeled toy vectors. The program calculates cosine scores, checks their order and demonstrates a zero-vector error. It is a math demonstration, not a production semantic retriever.
Expected interpretation
The toy vacation vector ranks above the unrelated vector because the numbers were selected that way. Real semantic relevance must be measured using a real embedding model and real queries. Write one case where an exact keyword should matter more than broad similarity.
Evaluate usefulness, not attractive scores
Record whether the top results contain the needed evidence, how often exact identifiers are missed, and the latency and cost of the complete pipeline. A cosine score is not a calibrated probability that a passage is correct.
A score needs context
A score of 0.8 in one embedding model cannot automatically be compared with 0.8 in another. Set thresholds from your own evaluation set and include unanswerable and misleadingly similar examples.
A practical workflow
- Explain the difference between a vector and the source text.
- Calculate cosine similarity for the supplied toy vectors.
- Inspect a retrieved passage for a fact, rather than trusting its ranking.
- Compare keyword and semantic retrieval on the same labeled questions.
Practice
Write three questions with different wording from a source, two exact-identifier questions and one unanswerable question. For each, name the expected source or state that no source supports an answer.
Check your understanding
Does a high similarity score prove that the answer is in the passage?
No. It ranks representational similarity under a chosen metric; inspect evidence coverage separately.
Are toy vector dimensions the meaning of real embedding dimensions?
No. The toy coordinates are manually chosen for arithmetic. Learned dimensions are not generally human-readable labels.
Can you freely mix embeddings from unrelated models?
No. Query and document representations must be compatible; follow the model’s documented modes and evaluate changes.
Sources
- Original RAG research — Lewis et al.: Research background on combining retrieval and generation. Examples and exercises in this guide are original teaching material.
- Sentence Transformers: semantic search: Primary documentation for further reading. The worked examples here are original teaching illustrations.
Continue learning
Read the original guide on Stack & Craft for the illustrated version and related lessons.
The Local RAG Document Assistant includes the companion exercises.
Top comments (0)