Search looks simple until you try to build a system that understands what people actually mean.
A traditional search engine sees:
"cheap laptops for developers"
and starts looking for documents containing:
cheap
laptops
developers
That works surprisingly well.
Until the user searches for:
"affordable machines for coding"
Now the problem becomes obvious.
The second query may be semantically almost identical to the first one, but the words are different.
A keyword search engine sees two different strings.
A semantic search engine sees two related ideas.
That difference is the foundation of modern intelligent search.
Semantic search attempts to answer a more interesting question:
Which documents are conceptually closest to what the user is asking for?
Instead of treating text as a collection of isolated words, we transform text into mathematical representations called embeddings.
Those embeddings allow us to represent meaning as vectors.
Then search becomes a geometric problem.
Text
│
▼
Embedding Model
│
▼
Vector
│
▼
┌────────────────────┐
│ Vector Database │
│ │
│ Similarity Search │
└─────────┬──────────┘
│
▼
Relevant Results
This article explores how to build semantic search from first principles and then turn it into something that looks much closer to a production backend system.
We will cover:
- embeddings
- vector spaces
- cosine similarity
- Euclidean distance
- nearest-neighbor search
- PostgreSQL + pgvector
- document chunking
- embedding pipelines
- metadata filtering
- hybrid search
- reranking
- indexing
- FastAPI implementation
- database schema design
- query optimization
- caching
- evaluation
- failure modes
- production architecture
- security
- and the deeper engineering principles behind semantic search.
The important part is not simply getting a vector database working.
The important part is understanding why semantic search works and where it fails.
1. The Problem With Keyword Search
Let's begin with a conventional search system.
Imagine a database containing articles:
1. Building APIs with Django
2. PostgreSQL indexing strategies
3. Machine learning for agriculture
4. Designing distributed systems
5. Securing REST APIs
A user searches:
"How do I protect my backend endpoints?"
A keyword engine might search for:
protect
backend
endpoints
But the document might say:
"Authentication and authorization are fundamental
components of secure API architecture."
There may be almost no exact word overlap.
Yet the document is clearly relevant.
This gives us the first fundamental distinction:
Keyword Search
query words
│
▼
exact / lexical matching
│
▼
relevant text
Semantic Search
query meaning
│
▼
vector representation
│
▼
semantic similarity
│
▼
relevant concepts
Keyword search asks:
"Which documents contain these words?"
Semantic search asks:
"Which documents mean something similar to this query?"
Neither is universally superior.
The strongest production search engines often combine both.
But before we combine them, we need to understand the mathematics.
2. What Is an Embedding?
An embedding is a numerical representation of some object.
For text, an embedding model converts a piece of text into a vector.
For example, imagine our model produces only three dimensions:
"backend development"
→ [0.82, 0.14, 0.61]
And:
"server-side programming"
→ [0.79, 0.17, 0.64]
While:
"banana farming"
→ [0.08, 0.91, 0.12]
The first two vectors are close together.
The third is far away.
farming
●
/
/
/
/
/
● backend
/
● server-side
The embedding model is effectively transforming language into geometry.
That is one of the most important ideas in modern machine learning.
3. Semantic Search Is Geometry
Suppose an embedding model generates vectors in:
[
\mathbb{R}^n
]
That means every document becomes a point in an (n)-dimensional space.
We cannot visualize thousands of dimensions directly.
But conceptually:
Dimension 2
↑
│
● │
│
│
● │
│
└────────────────→ Dimension 1
Similar concepts tend to occupy nearby regions of the embedding space.
The search problem becomes:
[
q = embedding(query)
]
Then:
[
d_i = embedding(document_i)
]
We want:
[
d_i \approx q
]
according to some similarity function.
This means semantic search is fundamentally:
Text
│
▼
Embedding
│
▼
Vector
│
▼
Distance / Similarity
│
▼
Nearest neighbors
│
▼
Results
That sounds simple.
The engineering becomes interesting when we have millions of vectors.
4. Why Embeddings Capture Meaning
An embedding model is trained to produce useful numerical representations.
It doesn't literally store a dictionary where:
"cat" = [....]
and:
"dog" = [....]
Instead, the model learns statistical relationships between language concepts.
As a result, related concepts can occupy similar regions of the vector space.
For example:
"software engineer"
"backend developer"
"programmer"
"application developer"
may have representations that are closer to each other than to:
"mountain climbing"
This creates a semantic topology.
Programming
●
● ●
backend frontend
● ●
● API
Agriculture
●
● ●
farming crops
The embedding model is effectively creating a coordinate system for meaning.
But there is an important caveat:
The coordinates themselves do not have human-readable meanings.
Dimension 42 does not necessarily mean "technicality."
Dimension 100 does not necessarily mean "agriculture."
The representation is distributed.
Meaning emerges from relationships between vectors.
5. The First Mathematical Primitive: Dot Product
Suppose we have two vectors:
[
A = [a_1, a_2, ..., a_n]
]
and:
[
B = [b_1, b_2, ..., b_n]
]
Their dot product is:
[
A \cdot B =
\sum_{i=1}^{n} a_i b_i
]
For example:
[
A = [1,2,3]
]
[
B = [4,5,6]
]
Then:
[
A \cdot B =
1(4)+2(5)+3(6)
]
[
=4+10+18
]
[
=32
]
The dot product can provide a measure of alignment.
But vector magnitude matters.
That leads us to cosine similarity.
6. Cosine Similarity
Cosine similarity measures the angle between two vectors.
The formula is:
[
cos(\theta)
\frac{A \cdot B}
{|A||B|}
]
where:
[
|A|
\sqrt{\sum a_i^2}
]
The important idea is that cosine similarity cares primarily about direction, not absolute magnitude.
Visualize it:
B
/
/
/
/
/ θ
/
/
●──────────────► A
If the vectors point in almost the same direction:
cos(θ) ≈ 1
If they are perpendicular:
cos(θ) ≈ 0
If they point in opposite directions:
cos(θ) ≈ -1
For many embedding systems, cosine similarity is a useful similarity measure.
7. Implementing Cosine Similarity
Python makes this easy:
import math
def cosine_similarity(a, b):
dot = sum(
x * y
for x, y in zip(a, b)
)
magnitude_a = math.sqrt(
sum(x * x for x in a)
)
magnitude_b = math.sqrt(
sum(x * x for x in b)
)
if magnitude_a == 0 or magnitude_b == 0:
return 0.0
return dot / (
magnitude_a * magnitude_b
)
Now:
a = [1, 2, 3]
b = [2, 4, 6]
print(cosine_similarity(a, b))
The result is approximately:
1.0
because the vectors point in exactly the same direction.
This tiny function demonstrates the mathematical core of many semantic search systems.
But it does not scale.
8. The Naive Search Algorithm
Suppose we have:
1 million documents
Each has an embedding.
A naive search does:
query
│
▼
embedding
│
▼
compare against document 1
│
▼
compare against document 2
│
▼
compare against document 3
│
▼
...
│
▼
compare against document 1,000,000
│
▼
sort
│
▼
top 10
Mathematically, this is approximately:
[
O(ND)
]
where:
- (N) = number of vectors
- (D) = vector dimension
For small datasets, this can be perfectly acceptable.
For large datasets, it becomes expensive.
This is where approximate nearest-neighbor indexing enters the picture.
9. Exact Search vs Approximate Search
There are two broad approaches.
Exact nearest-neighbor search
Compare the query against every vector.
Advantages:
- exact
- simple
- predictable
Disadvantages:
- expensive at scale
Approximate nearest-neighbor search
Use an index structure to avoid comparing against every vector.
Advantages:
- dramatically faster
- scales to large datasets
Disadvantages:
- results are approximate
- index tuning matters
The architecture becomes:
Exact Search
Query
│
├── Vector 1
├── Vector 2
├── Vector 3
├── Vector 4
├── ...
└── Vector N
│
▼
Top K
Approximate Search
Query
│
▼
Vector Index
│
├── candidate region
├── candidate region
└── candidate region
│
▼
Top K
This is one of the key engineering tradeoffs in semantic search.
10. PostgreSQL Can Become a Vector Database
If you're already building backend systems with PostgreSQL, you don't necessarily need to introduce a separate vector database immediately.
The pgvector extension allows PostgreSQL to store and search vector embeddings.
Conceptually:
PostgreSQL
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
relational JSONB vectors
data │
▼
semantic search
This is particularly attractive for applications where:
- documents already live in PostgreSQL
- users already live in PostgreSQL
- permissions already live in PostgreSQL
- metadata already lives in PostgreSQL
Instead of:
PostgreSQL
+
Vector Database
+
synchronization
you can initially use:
PostgreSQL
│
├── relational records
├── metadata
└── embeddings
Fewer systems means fewer synchronization problems.
11. Designing the Database Schema
Let's build a document table.
CREATE EXTENSION IF NOT EXISTS vector;
Then:
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
title TEXT NOT NULL,
content TEXT NOT NULL,
embedding VECTOR(1536),
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ
NOT NULL DEFAULT now()
);
The 1536 dimension here is only an example.
Your vector dimension must match the embedding model you actually use.
You can add metadata such as:
{
"category": "technology",
"author": "Derek",
"language": "en",
"visibility": "public"
}
This is important because semantic similarity alone is rarely enough.
12. Documents Should Usually Be Chunked
This is one of the most important practical decisions.
Suppose you have a 10,000-word article.
You could generate one embedding:
10,000-word article
│
▼
embedding
But now the vector represents the entire document.
A user might ask:
"How does the article explain PostgreSQL indexing?"
The answer could be buried in one paragraph.
The whole-document embedding may dilute that specific concept.
Instead, split the document into chunks.
Article
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
├── Chunk 4
├── Chunk 5
└── Chunk 6
Each chunk gets its own embedding.
Chunk 1 ──► Vector 1
Chunk 2 ──► Vector 2
Chunk 3 ──► Vector 3
...
Chunk 6 ──► Vector 6
Now search can identify the most relevant passage.
13. Chunking Is a Retrieval Problem
A common beginner mistake is:
"Split every 500 characters."
That's not necessarily good.
Imagine:
Chunk 1:
PostgreSQL indexes improve query performance...
Chunk 2:
...but only when the index matches the access pattern.
The second sentence depends on the first.
If we split them badly, we lose context.
Better chunking often respects:
- paragraphs
- headings
- sentences
- code blocks
- lists
- semantic sections
For technical documents, structure-aware chunking can be especially valuable.
A document might become:
Article
│
├── Introduction
│
├── What is PostgreSQL?
│
├── Indexing
│ ├── B-tree indexes
│ ├── Partial indexes
│ └── Composite indexes
│
├── Transactions
│
└── Conclusion
This structure gives us much better retrieval units.
14. Chunk Size Is a Tradeoff
Very small chunks:
Pros:
- precise retrieval
Cons:
- little context
- more vectors
- more indexing overhead
Very large chunks:
Pros:
- more context
- fewer vectors
Cons:
- diluted relevance
- larger retrieval payloads
The right chunk size depends on:
- document type
- embedding model
- query style
- application requirements
There is no universal magic number.
A practical strategy is to start with moderate chunks and measure retrieval quality.
15. Overlap Can Preserve Context
Suppose we create chunks of 500 tokens.
We might overlap by 50 tokens:
Chunk 1
████████████████████████████████
overlap
█████
███████████████████████████████
Chunk 2
The overlap helps prevent important information from being split exactly at a boundary.
Conceptually:
Document
A B C D E F G H I J K L M N
Chunk 1:
A B C D E F G
Chunk 2:
F G H I J K L M
Chunk 3:
L M N ...
The exact overlap should be measured rather than blindly chosen.
More overlap means:
- more vectors
- more storage
- more embedding cost
But it may improve retrieval quality.
16. Building a Chunking Function
A simple educational implementation:
def chunk_text(text, chunk_size=1000, overlap=200):
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunk = text[start:end]
chunks.append(chunk)
start = end - overlap
return chunks
This is intentionally simple.
A production system should preferably understand document structure rather than blindly slicing strings.
For example:
def structured_chunks(document):
sections = split_by_heading(document)
chunks = []
for section in sections:
chunks.extend(
split_by_semantic_boundary(section)
)
return chunks
The important idea is:
Chunking is part of search quality, not just preprocessing.
17. The Embedding Pipeline
Now we can build the ingestion pipeline.
Raw Document
│
▼
Clean Content
│
▼
Chunking
│
▼
Text Chunks
│
▼
Embedding Model
│
▼
Vector Embeddings
│
▼
PostgreSQL
│
▼
Vector Index
A document ingestion function might look like:
def index_document(document):
chunks = chunk_text(
document.content
)
for position, chunk in enumerate(chunks):
vector = embedding_model.embed(
chunk
)
save_chunk(
document_id=document.id,
position=position,
content=chunk,
embedding=vector
)
This is conceptually simple.
But production ingestion needs to deal with:
- retries
- rate limits
- partial failures
- duplicate documents
- model versioning
- deleted documents
- embedding migrations
- background processing
18. Separate Documents From Chunks
A better schema is often:
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
title TEXT NOT NULL,
source TEXT,
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
Then:
CREATE TABLE document_chunks (
id BIGSERIAL PRIMARY KEY,
document_id BIGINT NOT NULL
REFERENCES documents(id)
ON DELETE CASCADE,
chunk_index INTEGER NOT NULL,
content TEXT NOT NULL,
embedding VECTOR(1536),
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ
NOT NULL DEFAULT now(),
UNIQUE(document_id, chunk_index)
);
This gives us:
documents
│
├── chunk 0
├── chunk 1
├── chunk 2
└── chunk 3
Now search can return chunks while the application reconstructs the parent document.
19. Store Embedding Model Metadata
This is an underrated production concern.
Suppose you start with:
embedding-model-v1
Then six months later you migrate to:
embedding-model-v2
Your vector space may change.
You should know which model produced each vector.
For example:
ALTER TABLE document_chunks
ADD COLUMN embedding_model TEXT;
Then:
embedding_model = "model-v1"
This makes migrations much easier.
You may eventually need:
model-v1 vectors
model-v2 vectors
to coexist during migration.
20. Running a Semantic Query
Once vectors are stored, the search process becomes:
User Query
│
▼
Embedding Model
│
▼
Query Vector
│
▼
Vector Search
│
▼
Candidate Chunks
│
▼
Metadata Filters
│
▼
Reranking
│
▼
Final Results
The query vector is generated once.
Then PostgreSQL compares it against stored embeddings.
With pgvector, cosine distance can be expressed using the cosine-distance operator.
Conceptually:
SELECT
id,
document_id,
content,
embedding <=> :query_embedding AS distance
FROM document_chunks
ORDER BY embedding <=> :query_embedding
LIMIT 10;
Smaller distance means greater similarity.
You can convert distance into a similarity-like score:
1 - (embedding <=> :query_embedding)
depending on the metric and interpretation you are using.
21. Add an Index
For larger collections, you don't want to perform a full scan forever.
pgvector supports approximate nearest-neighbor indexing approaches such as HNSW and IVFFlat.
For example, an HNSW index can be created conceptually as:
CREATE INDEX document_chunks_embedding_idx
ON document_chunks
USING hnsw (embedding vector_cosine_ops);
Then:
Query Vector
│
▼
HNSW Index
│
▼
Nearest candidates
│
▼
Top K
The exact index choice and parameters should be benchmarked against your dataset.
A search engine is not finished when the query returns results.
It is finished when it returns good results quickly enough.
22. What Is HNSW?
HNSW stands for:
Hierarchical Navigable Small World.
It creates a graph structure that allows approximate nearest-neighbor navigation.
A simplified visualization:
Layer 2:
A ─────────────── D
\ /
\ /
─────── G ───
Layer 1:
A ── B ── C ── D ── E ── F ── G
\ \ / \ /
─────── H ───────────────
Instead of checking every vector, the search traverses the graph toward increasingly promising candidates.
Conceptually:
Query
│
▼
Start node
│
▼
Nearest neighbor
│
▼
Better neighbor
│
▼
Better neighbor
│
▼
Candidate set
This can dramatically reduce search work.
But HNSW introduces tradeoffs involving:
- memory
- index build time
- insertion cost
- search quality
- search latency
Again:
Benchmark instead of guessing.
23. Exact Search Is Still Valuable
Do not immediately assume approximate search is necessary.
If you have:
10,000 vectors
a sequential scan may be completely reasonable.
The architecture can be:
10k vectors
│
▼
exact cosine search
│
▼
fast enough
Introducing a complicated ANN index might add complexity without meaningful benefit.
This is a general engineering principle:
Optimize after measuring.
Semantic search is no exception.
24. Metadata Filtering Changes Everything
Suppose your database contains:
10 million documents
but the user should only search:
tenant_id = 42
You should not retrieve arbitrary documents and filter them in application code.
Instead:
SELECT
id,
document_id,
content
FROM document_chunks
WHERE metadata->>'tenant_id' = '42'
ORDER BY embedding <=> :query_embedding
LIMIT 10;
This is especially important in multi-tenant systems.
The search pipeline becomes:
User Query
│
▼
Query Embedding
│
▼
Tenant Filter
│
▼
Vector Search
│
▼
Top K
Security and relevance intersect here.
A semantically perfect result from the wrong tenant is still a security vulnerability.
25. Treat Authorization as Part of Retrieval
Imagine:
Company A
├── public documents
└── confidential documents
Company B
├── public documents
└── confidential documents
The search engine must never return Company A's private document to Company B.
Do not do:
search globally
│
▼
filter permissions later
if that can expose sensitive data through:
- result counts
- scores
- snippets
- logs
- caching
- timing
Instead, authorization constraints should participate in candidate retrieval.
Conceptually:
Query
│
├── tenant
├── user
├── roles
├── permissions
└── semantic vector
│
▼
Authorized candidate set
│
▼
Semantic ranking
This is one of the most important production lessons.
26. Semantic Search Alone Is Not Enough
Imagine the user searches:
"PostgreSQL"
They probably expect documents containing:
PostgreSQL
to rank highly.
But semantic search might retrieve:
relational database optimization
before:
PostgreSQL 18 release notes
because the semantic representation considers broader meaning.
This is why production systems often use hybrid search.
Combine:
lexical search
+
semantic search
27. Hybrid Search Architecture
A hybrid system might look like:
Query
│
┌────────┴────────┐
│ │
▼ ▼
Keyword Search Semantic Search
│ │
▼ ▼
BM25 results Vector results
│ │
└────────┬────────┘
▼
Result Fusion
│
▼
Reranker
│
▼
Final Results
Keyword search is good at:
- exact names
- IDs
- product codes
- technical terms
- rare words
Semantic search is good at:
- paraphrases
- conceptual similarity
- natural-language questions
- related concepts
Together they are much stronger.
28. BM25 + Vector Search
A common lexical ranking algorithm is BM25.
Conceptually:
BM25 score
+
semantic similarity
We can normalize the two scores and combine them:
[
Score =
\alpha \cdot SemanticScore
+
(1-\alpha)\cdot KeywordScore
]
For example:
[
Score =
0.7S_{semantic}
+
0.3S_{keyword}
]
This is only a starting point.
The correct weighting should be determined empirically.
29. Reciprocal Rank Fusion
Another approach is Reciprocal Rank Fusion.
Suppose:
Semantic ranking:
A
B
C
D
Keyword ranking:
C
A
D
B
RRF assigns a score based on rank.
Conceptually:
[
RRF(d)
\sum_i
\frac{1}{k + rank_i(d)}
]
This is useful because you don't necessarily need the scores from the two systems to be directly comparable.
You combine rankings instead.
Semantic results
│
▼
ranks
│
├──────────┐
│ │
Keyword results │
│ │
▼ │
ranks │
│ │
└────┬─────┘
▼
RRF
│
▼
unified ranking
30. Reranking
Initial retrieval might return:
top 50 candidates
Then a more expensive model reranks them.
Why?
Vector similarity is useful, but it is still a relatively coarse measure.
A reranker can examine:
query
+
candidate document
and produce a more precise relevance score.
The architecture becomes:
Query
│
▼
Embedding
│
▼
Retrieve 100 candidates
│
▼
Cheap ranking
│
▼
Reranker
│
▼
Top 10
This is a classic retrieval architecture.
You don't want to run an expensive reranker against millions of documents.
You use cheap retrieval first.
Then expensive reasoning on a small candidate set.
31. Search Is a Funnel
A useful mental model is:
1,000,000 documents
│
▼
Metadata filters
│
▼
100,000 candidates
│
▼
Vector retrieval
│
▼
100 candidates
│
▼
Reranking model
│
▼
10 results
│
▼
User interface
Every stage should reduce the search space.
This is how you make intelligent search affordable.
32. Building the Backend API
Let's build a simplified FastAPI endpoint.
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class SearchRequest(BaseModel):
query: str
limit: int = 10
@app.post("/search")
def search(request: SearchRequest):
query_vector = embedding_model.embed(
request.query
)
results = search_database(
query_vector,
request.limit
)
return {
"query": request.query,
"results": results
}
The backend flow is:
POST /search
│
▼
validate request
│
▼
generate embedding
│
▼
query PostgreSQL
│
▼
retrieve candidates
│
▼
return results
But this simple endpoint hides several important engineering problems.
33. Embedding Generation Is Often the Expensive Part
Suppose your search endpoint receives:
100 requests/second
and every request requires an embedding API call.
Now you have:
100 embedding requests/sec
That introduces:
- latency
- cost
- rate limits
- external dependency
You can mitigate this with caching.
For example:
query
│
▼
normalize
│
▼
hash(query)
│
▼
cache lookup
│
├── hit ─────► vector
│
└── miss ────► embedding model
│
▼
cache
A simple cache key might be:
semantic-search:v1:<hash>
The model version belongs in the cache key.
Otherwise:
model-v1 query vector
might accidentally be reused after migrating to:
model-v2
34. Query Normalization
Before embedding, you can normalize certain query forms.
For example:
"How do I build an API?"
and:
"how do i build an api?"
may be equivalent.
But be careful.
Aggressive normalization can destroy meaning.
For example:
"C++"
must not become:
"c"
And:
"Node.js"
should not become:
"node"
Search normalization should be driven by evaluation.
Don't normalize because it "looks cleaner."
Normalize because it improves retrieval.
35. Semantic Search Over Technical Documentation
Let's consider a real backend use case.
Suppose we have documentation:
PostgreSQL Tricks
Docker Deployment Guide
Django Authentication
React State Management
Kubernetes Networking
The user asks:
"How do I stop duplicate records from being created?"
The PostgreSQL article may contain:
UNIQUE constraints
ON CONFLICT
idempotency
There may be no exact phrase:
"stop duplicate records"
But semantic retrieval can connect:
duplicate records
│
▼
idempotency
│
▼
ON CONFLICT
│
▼
PostgreSQL chunk
This is where semantic search becomes much more useful than a basic SQL LIKE.
36. Semantic Search Is Not Magic
Suppose the user searches:
"error code 0x80131500"
Semantic search might not be the best tool.
The exact identifier matters.
Similarly:
"invoice 847291"
requires lexical precision.
Or:
"API-KEY-ABC123"
is not a semantic concept.
This is why hybrid search is so powerful.
Exact identifiers
│
▼
Keyword search
Natural language concepts
│
▼
Semantic search
The right search engine uses the right retrieval mechanism for the query.
37. Build a Search Class
A cleaner architecture:
class SemanticSearch:
def __init__(
self,
embedder,
repository,
reranker=None
):
self.embedder = embedder
self.repository = repository
self.reranker = reranker
def search(
self,
query,
limit=10
):
vector = self.embedder.embed(
query
)
candidates = self.repository.vector_search(
vector,
limit=50
)
if self.reranker:
candidates = self.reranker.rank(
query,
candidates
)
return candidates[:limit]
This separates:
embedding
retrieval
reranking
That separation becomes very useful as the system grows.
38. Repository Layer
Your database code should not be scattered through API controllers.
For example:
class DocumentRepository:
def vector_search(
self,
vector,
limit=10
):
sql = """
SELECT
id,
document_id,
content,
1 - (embedding <=> %s)
AS similarity
FROM document_chunks
ORDER BY embedding <=> %s
LIMIT %s
"""
return execute(
sql,
[vector, vector, limit]
)
Now your application architecture becomes:
FastAPI
│
▼
Search Service
│
├── Embedding Provider
│
├── Repository
│
└── Reranker
│
▼
PostgreSQL
This keeps infrastructure replaceable.
39. Background Indexing
Don't generate embeddings synchronously during every document upload if the process is expensive.
Instead:
POST /documents
│
▼
store document
│
▼
create indexing job
│
▼
return 202
│
▼
background worker
│
├── chunk
├── embed
└── store vectors
This makes the user-facing API faster.
A PostgreSQL-backed job queue can even work for smaller systems.
documents
│
▼
embedding_jobs
│
▼
worker
│
▼
document_chunks
The document becomes searchable once indexing completes.
40. Idempotent Indexing
Suppose the worker crashes halfway through indexing.
You don't want:
document 42
├── chunk 1
├── chunk 2
├── chunk 3
├── chunk 3
├── chunk 4
└── chunk 4
Use a unique constraint:
UNIQUE(document_id, chunk_index)
Then indexing can safely use:
INSERT INTO document_chunks (...)
VALUES (...)
ON CONFLICT (
document_id,
chunk_index
)
DO UPDATE SET
content = EXCLUDED.content,
embedding = EXCLUDED.embedding;
Now retries become safe.
This is the same backend engineering principle we use everywhere:
Make expensive operations idempotent.
41. Versioning the Index
Suppose your chunking strategy changes.
Version 1:
chunk_size = 500
Version 2:
chunk_size = 800
Your embedding model changes.
Your metadata extraction changes.
Your preprocessing changes.
You need to know which pipeline generated which vector.
Add:
embedding_version TEXT NOT NULL
and perhaps:
chunking_version TEXT NOT NULL
Now you can reason about your search corpus.
Document
│
├── pipeline v1
│ └── vectors
│
└── pipeline v2
└── vectors
This is essential for controlled migrations.
42. Evaluation Is More Important Than the Demo
A semantic search demo can look incredible.
You enter:
"How do I secure my API?"
and get:
API Security
Perfect.
But production search has thousands of queries.
You need evaluation.
Create a dataset:
query relevant_docs
"secure API endpoints" [12, 44]
"postgres duplicate records" [8]
"docker deployment" [19, 21]
"database transaction" [4, 7]
Then measure retrieval quality.
43. Precision and Recall
Precision asks:
Of the documents returned, how many are relevant?
[
Precision =
\frac{Relevant\ Retrieved}
{Retrieved}
]
Recall asks:
Of all relevant documents, how many did we retrieve?
[
Recall =
\frac{Relevant\ Retrieved}
{Relevant}
]
For example:
Retrieved = 10
Relevant retrieved = 8
Then:
[
Precision = 0.8
]
If there were 20 relevant documents overall:
[
Recall = 8/20 = 0.4
]
This tells you something important.
Your system can be precise but incomplete.
Or broad but noisy.
44. Top-K Metrics
Search systems often care about:
Precision@K
Recall@K
MRR
NDCG
For example:
Precision@5
asks:
How many of the first five results are relevant?
This is often more meaningful than overall precision because users rarely inspect result 200.
A search system should optimize for the top of the ranking.
45. Human Evaluation Still Matters
Metrics are useful.
But search quality is ultimately about users.
Give real users queries and ask:
Was result #1 useful?
Was result #2 useful?
Did the system answer your intent?
Did you find what you wanted?
You might discover:
Vector similarity = excellent
User satisfaction = poor
Why?
Maybe the results are semantically similar but not actionable.
This is why semantic search should be evaluated as a product feature, not merely an ML benchmark.
46. Common Failure: Bad Chunking
Suppose a document says:
PostgreSQL uses MVCC.
This means transactions can operate concurrently
without readers blocking writers in many cases.
If chunking produces:
Chunk 1:
PostgreSQL uses MVCC.
Chunk 2:
This means transactions...
the second chunk may lose important context.
Better:
Chunk 1:
PostgreSQL uses MVCC. This means transactions
can operate concurrently without readers blocking
writers in many cases.
The embedding now captures the concept more completely.
Search quality often improves more from better chunking than from blindly switching models.
47. Common Failure: Embedding the Wrong Content
Suppose a product database contains:
name
price
stock
description
category
You generate embeddings from:
"19.99"
That is useless.
Instead:
Product:
Mechanical Keyboard
Category:
Computer Accessories
Description:
Compact mechanical keyboard designed for
software developers and gamers...
Then embed the semantic content.
You can separately store structured fields for filtering.
Embedding:
meaning
Metadata:
price
category
availability
tenant
permissions
This separation is powerful.
48. Structured Data and Semantic Data Should Coexist
Imagine an e-commerce search:
"black laptop for programming under $1500"
The semantic part is:
laptop
programming
The structured constraints are:
color = black
price <= 1500
A good search architecture does:
Natural language query
│
├───────────────┐
│ │
▼ ▼
semantic concepts structured filters
│ │
└───────┬───────┘
▼
candidate search
│
▼
ranking
This is one of the most important patterns for production search.
49. Query Understanding
You can explicitly extract structured constraints.
For example:
"black laptop for programming under $1500"
becomes:
{
"semantic_query": "laptop for programming",
"filters": {
"color": "black",
"price_max": 1500
}
}
Then:
semantic_query
│
▼
embedding
│
▼
vector search
filters
│
▼
SQL predicates
This produces much better results than trying to make one vector represent everything.
50. Semantic Search and RAG
Semantic search is also one of the foundations of Retrieval-Augmented Generation.
The architecture looks like:
User Question
│
▼
Query Embedding
│
▼
Semantic Retrieval
│
▼
Relevant Chunks
│
▼
Context Window
│
▼
Language Model
│
▼
Generated Answer
This is where semantic search becomes particularly powerful.
Instead of asking a language model to know everything, we retrieve relevant knowledge from our own corpus.
For example:
Question:
"What is our company's refund policy?"
Semantic search retrieves:
Refund Policy
Section 4
Then the model uses that content to formulate the answer.
51. But RAG Quality Is Search Quality
A common misconception is:
"If the language model is powerful enough, it will fix bad retrieval."
Usually it won't.
If retrieval gives the model:
wrong documents
the model has limited ability to magically discover the correct source.
Therefore:
RAG quality
≈
retrieval quality
+
generation quality
If retrieval is broken, generation can become confidently wrong.
This is why search engineering is becoming increasingly important in AI systems.
52. Security in Semantic Search
Vector databases contain data representations derived from your documents.
Those vectors are not automatically harmless.
If your corpus contains:
private customer data
internal documents
financial information
API documentation
credentials
your vector index becomes part of your sensitive data infrastructure.
Security must include:
- tenant isolation
- authentication
- authorization
- encryption at rest
- encryption in transit
- audit logging
- access controls
- deletion workflows
- retention policies
And don't forget the source documents.
Deleting a document should also remove:
document
chunks
embeddings
cache entries
search indexes
derived artifacts
Data deletion becomes a pipeline.
53. The Deletion Problem
Suppose a customer asks:
"Delete all my data."
You remove the document from the primary table.
But perhaps you still have:
document_chunks
embedding cache
search cache
analytics logs
backup
vector index
The search system can accidentally continue returning deleted information.
Therefore, deletion must propagate.
Delete document
│
├── delete chunks
│
├── invalidate caches
│
├── remove search references
│
└── schedule backup lifecycle
This is another reason to keep the architecture understandable.
54. Caching Search Results
Caching can exist at several layers.
Query embedding cache
query
│
▼
embedding cache
│
▼
vector
Search result cache
query + filters + model_version
│
▼
cache
│
▼
results
Document cache
chunk ID
│
▼
content cache
But cache keys must include all relevant parameters.
For example:
tenant_id
query
filters
model_version
index_version
top_k
Otherwise, one user's results might accidentally be returned to another user.
That's not a performance bug.
That's a security incident.
55. Observability
A production semantic search system should measure:
embedding latency
database latency
reranking latency
total latency
cache hit rate
result count
score distribution
zero-result rate
click-through rate
A useful trace might look like:
/search
│
├── embedding: 72ms
│
├── PostgreSQL: 18ms
│
├── reranker: 110ms
│
└── total: 205ms
Now you know where the bottleneck is.
Without observability, you are guessing.
56. Score Distributions Matter
Suppose every query produces:
Result 1: 0.91
Result 2: 0.90
Result 3: 0.89
That may indicate strong semantic matches.
But another query might produce:
Result 1: 0.42
Result 2: 0.41
Result 3: 0.40
Should you still return them?
Maybe.
Maybe not.
A fixed:
top_k = 10
doesn't guarantee relevance.
Sometimes you need a threshold:
if similarity < 0.55:
don't return
But the correct threshold depends on:
- embedding model
- corpus
- query distribution
- metric
- normalization
Again, evaluate empirically.
57. Semantic Search Is a Ranking System
At its core, the search engine is producing:
(query, document) → score
Then:
sort(document, score)
But production systems often have multiple signals:
semantic similarity
keyword relevance
freshness
authority
popularity
permissions
business rules
user preferences
A more realistic ranking function might be:
[
Score =
w_1 Semantic
+
w_2 Keyword
+
w_3 Freshness
+
w_4 Authority
+
w_5 Popularity
]
The weights are learned or tuned from evaluation data.
This is where search starts becoming information retrieval engineering rather than simply vector similarity.
58. Freshness Can Matter
Imagine searching:
"latest PostgreSQL security update"
An old article may be semantically perfect.
But it is not useful if the user needs current information.
Therefore:
semantic relevance
+
recency
can be more useful.
For example:
[
FinalScore =
0.8 \cdot SemanticScore
+
0.2 \cdot FreshnessScore
]
The correct weighting depends on the application.
For legal, financial, security, and technical content, freshness can be especially important.
59. Popularity Can Be Dangerous
Suppose your system always boosts popular documents.
Then:
popular
can overpower:
relevant
This creates a feedback loop:
popular document
│
▼
higher ranking
│
▼
more clicks
│
▼
more popularity
│
└───────────────┐
▼
even higher rank
Search ranking should be carefully designed to avoid reinforcing irrelevant content.
60. Build a Production Architecture
Let's put everything together.
┌──────────────────┐
│ Client │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ API │
│ FastAPI │
└────────┬─────────┘
│
┌─────────────┴────────────┐
│ │
▼ ▼
Query Service Auth Service
│
▼
Query Embedding
│
▼
┌──────────────────┐
│ Retrieval Layer │
└────────┬─────────┘
│
┌──────────┴──────────┐
│ │
▼ ▼
PostgreSQL Lexical Search
+ pgvector │
│ │
└──────────┬──────────┘
▼
Candidate Merge
│
▼
Reranker
│
▼
Final Ranking
│
▼
Cache
│
▼
Response
The ingestion pipeline is separate:
Documents
│
▼
Ingestion API
│
▼
Job Queue
│
▼
Workers
│
├── parse
├── clean
├── chunk
├── embed
└── persist
│
▼
PostgreSQL
This separation is important.
Search should not be blocked by ingestion.
61. A More Complete PostgreSQL Schema
Here is a practical starting point:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
tenant_id BIGINT NOT NULL,
title TEXT NOT NULL,
source TEXT,
metadata JSONB NOT NULL DEFAULT '{}',
content_hash TEXT NOT NULL,
created_at TIMESTAMPTZ
NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ
NOT NULL DEFAULT now(),
UNIQUE (
tenant_id,
content_hash
)
);
CREATE TABLE document_chunks (
id BIGSERIAL PRIMARY KEY,
document_id BIGINT NOT NULL
REFERENCES documents(id)
ON DELETE CASCADE,
chunk_index INTEGER NOT NULL,
content TEXT NOT NULL,
embedding VECTOR(1536),
embedding_model TEXT NOT NULL,
chunking_version TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ
NOT NULL DEFAULT now(),
UNIQUE (
document_id,
chunk_index
)
);
Then:
CREATE INDEX idx_document_chunks_document
ON document_chunks(document_id);
And:
CREATE INDEX idx_document_chunks_embedding
ON document_chunks
USING hnsw (embedding vector_cosine_ops);
The exact vector dimension and index configuration should match your selected embedding model and workload.
62. A Complete Search Query
Suppose we want tenant-specific search.
SELECT
c.id,
c.document_id,
c.content,
d.title,
1 - (
c.embedding <=> :query_embedding
) AS similarity
FROM document_chunks c
JOIN documents d
ON d.id = c.document_id
WHERE d.tenant_id = :tenant_id
ORDER BY c.embedding <=> :query_embedding
LIMIT :limit;
This is already a useful semantic search backend.
But production systems often go further:
tenant filter
+
permission filter
+
semantic search
+
keyword search
+
reranking
+
freshness
The vector query is only one stage.
63. Search API Response Design
Don't just return:
[
{
"content": "..."
}
]
Return useful metadata.
For example:
{
"query": "how do I prevent duplicate records?",
"results": [
{
"document_id": 42,
"chunk_id": 801,
"title": "PostgreSQL Tricks",
"content": "Use ON CONFLICT...",
"score": 0.91,
"metadata": {
"category": "database"
}
}
]
}
This allows the frontend to show:
PostgreSQL Tricks
Use ON CONFLICT to safely handle
duplicate insert attempts...
Similarity: 0.91
But be careful about exposing raw similarity scores to users.
They are often meaningful internally but not necessarily intuitive as user-facing confidence values.
64. Search Highlighting Still Matters
Even semantic search benefits from highlighting.
Suppose the result is:
PostgreSQL supports ON CONFLICT...
The UI could show:
PostgreSQL supports **ON CONFLICT**
for safely handling duplicate records...
This helps users understand why the result was returned.
Semantic ranking tells us:
"This is probably relevant."
Highlighting tells the user:
"Here is the part that might matter."
These are different jobs.
65. The Danger of Hallucinated Relevance
A semantic search engine can return a result that feels related but does not actually answer the query.
Example:
Query:
"How do I configure PostgreSQL replication?"
Result:
"PostgreSQL backup strategies"
Related?
Yes.
Answer?
Not necessarily.
This is why reranking and evaluation matter.
Semantic similarity is not the same thing as usefulness.
66. Build Search Around User Intent
A better system tries to identify intent.
For example:
"How do I configure replication?"
Intent:
technical how-to
Query:
"PostgreSQL replication"
Another:
"Why is my PostgreSQL query slow?"
Intent:
troubleshooting
The retrieval system might prioritize:
EXPLAIN ANALYZE
indexes
query plans
locks
This is where search becomes more intelligent than simply comparing embeddings.
67. Semantic Search and Backend Engineering
Semantic search is an interesting intersection of several disciplines.
Semantic Search
│
┌──────────────┼───────────────┐
│ │ │
▼ ▼ ▼
Information Machine Backend
Retrieval Learning Engineering
│ │ │
▼ ▼ ▼
ranking embeddings APIs
indexing models databases
precision inference caching
recall evaluation concurrency
This is why building semantic search is such a useful backend project.
You learn machine learning without needing to train a foundation model.
You learn databases without abandoning application engineering.
And you learn search without having to build Google.
68. The Most Important Optimization: Reduce Work
Suppose you have:
100 million vectors
Your biggest optimization is not:
make cosine similarity 10% faster
It is:
avoid comparing against 99.99% of vectors
This leads to:
filter
→ retrieve
→ rerank
rather than:
compare everything
→ sort everything
Systems engineering is often about avoiding unnecessary work.
Semantic search makes this principle very visible.
69. When You Should Use a Dedicated Vector Database
PostgreSQL + pgvector is excellent for many applications.
But eventually you may need specialized infrastructure.
Consider a dedicated vector system when you have requirements around:
- enormous vector collections
- specialized ANN workloads
- very high search throughput
- distributed vector indexing
- specialized filtering and retrieval features
- independent scaling of vector search
The architecture can evolve:
Early stage
PostgreSQL
├── documents
├── metadata
└── vectors
Later stage
PostgreSQL
│
├── authoritative data
│
└── metadata
Vector system
│
└── embeddings
But introducing another database creates synchronization problems.
Therefore:
Don't introduce a vector database because semantic search is fashionable.
Introduce it because your workload requires it.
70. Database First, Search Second
This is a principle I like for backend architecture.
Your authoritative data should remain authoritative.
The embedding is a derived representation.
Source document
│
▼
embedding pipeline
│
▼
vector
If the vector disappears:
rebuild
You should not lose the underlying document.
This makes embeddings similar to:
cache
index
materialized view
derived state
That mental model is extremely useful.
71. Embeddings Are Derived State
Think of the relationship:
Document
│
├── title
├── content
└── metadata
│
▼
embedding model
│
▼
embedding
If you change:
embedding model
you should be able to regenerate the vectors.
If you change:
chunking strategy
you should be able to regenerate the chunks.
This means your pipeline should be reproducible.
That is a very important production requirement.
72. Reproducibility
Store:
document hash
embedding model
embedding version
chunking version
preprocessing version
Then you can answer:
"Why does this document have this vector?"
You should be able to trace:
document
│
▼
content hash
│
▼
preprocessing v3
│
▼
chunking v2
│
▼
embedding-model-v4
│
▼
vector
This is search observability at the data level.
73. Semantic Search Is a Pipeline, Not a Feature
It is tempting to describe semantic search as:
vector database
That is incomplete.
A real system is:
┌─────────────┐
│ Sources │
└──────┬──────┘
│
▼
┌─────────────┐
│ Parsing │
└──────┬──────┘
│
▼
┌─────────────┐
│ Chunking │
└──────┬──────┘
│
▼
┌─────────────┐
│ Embeddings │
└──────┬──────┘
│
▼
┌─────────────┐
│ Vector DB │
└──────┬──────┘
│
▼
Retrieval
│
▼
Ranking
│
▼
Reranking
│
▼
Results
If any stage is poor, the user experience suffers.
74. A Practical Development Roadmap
If I were building semantic search from scratch, I would not begin with a huge architecture.
I would build it in stages.
Stage 1
PostgreSQL
+
pgvector
+
embedding model
Implement:
document ingestion
chunking
embedding
vector search
Stage 2
Add:
metadata filtering
caching
background workers
Stage 3
Add:
keyword search
hybrid retrieval
Stage 4
Add:
reranking
Stage 5
Build:
evaluation dataset
metrics
observability
Stage 6
Optimize:
ANN indexes
query latency
embedding costs
cache hit rates
Stage 7
Only then consider:
dedicated vector infrastructure
This progression keeps the architecture understandable.
75. Final Mental Model
If you remember only one diagram from this article, remember this:
USER
│
▼
QUERY
│
▼
Query Understanding
│
┌──────────┴──────────┐
│ │
▼ ▼
Keywords Embedding
│ │
▼ ▼
Lexical Search Vector Search
│ │
└──────────┬──────────┘
▼
Candidates
│
▼
Reranking
│
▼
Business Rules
│
▼
Final Results
│
▼
USER
And the data pipeline:
DOCUMENT
│
▼
CLEANING
│
▼
CHUNKING
│
▼
EMBEDDING
│
▼
VECTOR + METADATA
│
▼
VECTOR INDEX
│
▼
RETRIEVAL
These two pipelines form the heart of semantic search.
Final Thoughts
Semantic search looks like an AI problem.
It is.
But it is also a database problem.
It is also an information retrieval problem.
It is also a ranking problem.
And, perhaps most importantly for backend developers, it is a systems engineering problem.
The first breakthrough is understanding that text can be represented as vectors.
The second is realizing that vectors create geometry.
The third is realizing that search can become nearest-neighbor retrieval.
Then the real engineering begins.
You need to decide how documents are chunked.
You need to decide how embeddings are generated.
You need to store them efficiently.
You need to retrieve them quickly.
You need to filter by permissions.
You need to combine semantic and lexical signals.
You need to rerank candidates.
You need to evaluate relevance.
You need to cache expensive operations.
You need to monitor latency.
You need to version your embedding pipeline.
You need to handle deletion.
You need to prevent tenant leakage.
And you need to know when semantic search is simply the wrong tool.
The most important lesson is this:
An embedding is not search.
An embedding is a representation.
The search engine is the system built around that representation.
That distinction changes how you architect the entire application.
A weak implementation says:
text
↓
embedding
↓
database
↓
results
A mature implementation looks more like:
Documents
│
▼
Parsing / Cleaning
│
▼
Chunking
│
▼
Embeddings
│
▼
PostgreSQL + pgvector
│
▼
Query Embedding
│
┌───────────┴───────────┐
│ │
▼ ▼
Lexical Search Vector Search
│ │
└───────────┬───────────┘
▼
Candidate Set
│
▼
Reranking
│
▼
Authorization Rules
│
▼
Final Results
And this architecture reveals something deeper.
Search is fundamentally about reducing uncertainty.
The user has a question.
The system has millions of possible documents.
The job of the search engine is to progressively narrow that enormous space until a small number of highly relevant pieces of information remain.
Keyword search narrows the space through words.
Semantic search narrows it through meaning.
Hybrid search uses both.
Reranking applies deeper judgment.
Metadata filters apply hard constraints.
And the final ranking becomes the interface between an enormous information space and a human being.
That is why semantic search is such an interesting engineering problem.
You are not simply storing vectors.
You are building a system that tries to understand what a user means, map that meaning into mathematical space, navigate an enormous collection of representations, apply business constraints, rank competing interpretations, and return useful information fast enough that the user never notices the complexity underneath.
The code may eventually look surprisingly small.
The architecture behind it is not.
And that is the real lesson of building semantic search:
The difficult part isn't generating the vector.
The difficult part is turning a vector into a trustworthy search experience.
Top comments (0)