In the previous post, we looked at tokens — the units language models use to process text. But tokens alone don't carry meaning. To a model, they're just IDs.
That's where embeddings come in.
Embeddings are one of the most important ideas in modern AI. They power semantic search, RAG systems, recommendation engines, AI memory, code search, and much more. If you've ever searched for something and found relevant results even when the exact words didn't match, embeddings were probably involved.
Let's break down how they work.
The Problem: Computers Don't Understand Meaning
Humans naturally understand that these ideas are related:
- Dog
- Puppy
- Golden Retriever
And we instantly know they're unrelated to:
- PostgreSQL sharding
- Redis caching
- Kafka partitions
Computers don't have that intuition.
Traditional software only sees text as symbols. It can match exact words, but it doesn't inherently understand relationships between concepts.
That's a problem because people rarely search using the exact words that appear in documents.
Imagine a user searches:
How do I scale PostgreSQL?
Your documentation contains:
- PostgreSQL Horizontal Scaling
- Database Sharding Strategies
- PgBouncer Connection Pooling
- Scaling Postgres Databases
A keyword search might miss some of the most useful results simply because the wording differs.
Humans see the connection immediately.
Computers need a way to represent meaning mathematically.
What Is an Embedding?
An embedding is a vector of numbers that represents the meaning of some input.
That input could be:
- Text
- Code
- Images
- Audio
- Documents
A simple example might look like:
"I love pizza"
↓
[0.12, -0.44, 0.89, 0.31, -0.72, ...]
Real embeddings are much larger.
For example:
| Model | Dimensions |
|---|---|
| OpenAI text-embedding-3-small | 1536 |
| OpenAI text-embedding-3-large | 3072 |
| Google text-embedding-004 | 768 |
| Sentence-BERT (all-MiniLM-L6) | 384 |
The important thing isn't any individual number.
The meaning comes from the entire pattern across the vector.
Think of it like GPS coordinates.
A single coordinate doesn't tell you much. Together, all coordinates define a location.
Embeddings do the same thing for ideas.
From Tokens to Meaning
In the previous article, we saw that tokenization converts text into token IDs.
"The quick brown fox"
↓
[791, 4062, 14198, 39935]
Those IDs contain no semantic meaning.
An embedding model takes those token representations and transforms them into a dense vector that captures relationships, context, and meaning.
Conceptually:
Text
↓
Tokens
↓
Token IDs
↓
Embedding Model
↓
Vector Representation
The result is a numerical representation where similar concepts end up near each other.
The Magic: Similar Ideas End Up Close Together
This is the key property that makes embeddings useful.
Suppose we embed thousands of technical documents.
Documents about:
- PostgreSQL
- Database scaling
- Replication
- Sharding
naturally cluster together.
Documents about:
- Redis
- Caching
- CDN strategies
form another cluster.
Documents about:
- Recipes
- Pizza
- Baking
end up somewhere completely different.
The model was never explicitly told:
PostgreSQL and sharding are related.
It learned those relationships from patterns in training data.
As a result, semantic similarity becomes geometric distance.
In embedding space:
- Near = similar meaning
- Far = unrelated meaning
That's the entire foundation of semantic search.
Finding Similar Content
Once everything is represented as vectors, search becomes a geometry problem.
Instead of asking:
Which documents contain these exact words?
we ask:
Which vectors are closest to this vector?
When a user searches:
How do I scale PostgreSQL?
the query is converted into an embedding.
Then the system searches for nearby vectors among stored documents.
The closest vectors are usually the most relevant results.
Even if none of those documents contain the exact phrase "scale PostgreSQL."
This is why semantic search feels much smarter than keyword search.
How Similarity Is Measured
To compare embeddings, we need a distance metric.
The most common approaches are:
| Method | Measures | Common Use |
|---|---|---|
| Cosine Similarity | Angle between vectors | Text similarity |
| Dot Product | Direction + magnitude | Normalized embeddings |
| Euclidean Distance | Straight-line distance | Images and some multimodal systems |
In practice, cosine similarity is the most widely used for text embeddings.
It focuses on whether vectors point in the same direction rather than how large they are.
That's useful because a short document and a long document about the same topic should still be considered similar.
Why Embeddings Matter for RAG
Embeddings are one of the core building blocks of Retrieval-Augmented Generation (RAG).
The workflow looks like this:
User Question
↓
Create Query Embedding
↓
Search Vector Database
↓
Retrieve Relevant Chunks
↓
Send Context To LLM
↓
Generate Response
Without embeddings, retrieval would mostly rely on keyword matching.
With embeddings, retrieval becomes semantic.
That's why RAG systems can often find relevant information even when users don't use the same terminology as the source documents.
Real-World Applications
Embeddings quietly power many products people use every day.
Semantic Search
Instead of matching words, search systems match meaning.
Examples:
- Enterprise knowledge bases
- Documentation search
- Internal company search
RAG Systems
Knowledge is embedded, stored, retrieved, and injected into LLM prompts.
Examples:
- AI chatbots
- Customer support assistants
- Internal copilots
Recommendation Engines
Products, movies, songs, and articles can all be represented as vectors.
Examples:
- Netflix recommendations
- Spotify music discovery
- Amazon product suggestions
AI Memory
Conversations can be embedded and stored.
Later, the system retrieves semantically relevant memories rather than searching exact text.
Code Search and Copilots
Tools like GitHub Copilot use embeddings to find relevant code context before generating suggestions.
Common Misconceptions
"Embeddings store the original text."
They don't.
Embeddings store a numerical representation of meaning, not the original text itself.
You generally cannot reconstruct the original document from an embedding vector.
"Nearby vectors mean identical meaning."
Not necessarily.
Nearby vectors indicate similar meaning.
Two documents can be closely related without being identical.
"Embeddings understand concepts like humans."
Not really.
Embeddings capture statistical patterns from training data.
They don't possess human understanding.
"More dimensions always means better embeddings."
Not always.
A well-trained 384-dimensional model can outperform a poorly trained 3000-dimensional model.
Training quality matters more than raw dimensionality.
"Embeddings are unbiased."
They aren't.
Embeddings inherit patterns and biases from the data they were trained on.
Scaling to Billions of Vectors
A small demo might compare a query against a few thousand vectors.
Production systems often store millions or billions.
Comparing every vector one by one becomes too slow.
That's why vector databases use Approximate Nearest Neighbour (ANN) algorithms such as:
- HNSW
- IVF
- PQ
Popular vector databases include:
- Pinecone
- Weaviate
- Qdrant
- pgvector
- Milvus
These systems can search enormous vector collections in milliseconds.
Key Takeaways
- Embeddings convert meaning into vectors of numbers.
- Similar meanings become geometrically close in embedding space.
- Semantic search works by finding nearby vectors.
- RAG systems depend on embeddings for retrieval.
- Cosine similarity is the most common way to compare text embeddings.
- Vector databases make large-scale similarity search practical.
What's Next?
Embeddings solve one problem:
How do we represent meaning as vectors?
The next challenge is:
How do we efficiently search billions of those vectors in milliseconds?
That's where vector databases come in.
If you've ever used a RAG system, AI memory feature, semantic search engine, or recommendation system, a vector database was probably working behind the scenes.
I also created an interactive visual walkthrough of embeddings and semantic search on SeeItFlow:
https://seeitflow.com/ai/ai-foundations/embeddings-explained
I'm curious — when did embeddings finally click for you?
Was it semantic search, RAG, recommendation systems, or something else?
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.