Vector Database Engineering
A deep-dive walkthrough of how vector databases work and how to run them well — covering vector dimensions and similarity metrics (cosine vs. dot product vs. Euclidean), approximate nearest-neighbor search, HNSW and IVF indexing internals, indexing with filters, hybrid indexes, multi-tenancy patterns, the vector data lifecycle, embedding versioning, migrations, performance tuning, and security — with C# and SQL code, toy index implementations that make the algorithms concrete, capacity-planning calculators, and interview-style questions.
Table of Contents
- Introduction
- Vector Dimensions and Similarity Metrics
- Cosine Similarity vs. Dot Product vs. Euclidean Distance
- Approximate Nearest-Neighbor Search
- HNSW Indexing
- IVF Indexing Concepts
- Vector Indexing and Filtering
- Hybrid Indexes
- Multi-Tenancy
- Vector Data Lifecycle
- Embedding Versioning
- Vector Database Migrations
- Vector Search Performance Tuning
- Vector Search Security
- Capacity Planning
- Choosing and Operating a Vector Database
- Interview-Style Questions
- Common Pitfalls
- Quick Reference Table
- Conclusion
Introduction
The Embeddings & Vector Search guide in this series showed how to use vector search: embed text, store vectors, query for neighbors, and combine with keywords. The Advanced RAG Architecture guide showed what to build on top of it. This guide goes one layer down and asks the engineering questions that determine whether a vector store is fast, accurate, affordable, and safe at production scale: What do the dimensions and metrics actually imply? How do HNSW and IVF trade recall for speed — and what do their parameters really do? Why does a filter make approximate search worse? How do you isolate tenants? What happens to the index when documents change or are deleted? How do you change embedding models without an outage? How do you move to another engine? What do you tune, and how do you know it worked? What can an attacker do with a vector database?
The shape of the problem:
N vectors x d dimensions -> exact search costs O(N x d) per query -> too slow at scale
-> APPROXIMATE indexes (HNSW graphs, IVF partitions, quantization) trade a little RECALL for a lot of SPEED and MEMORY
-> but real queries have FILTERS (tenant, ACL, type), CHURN (updates, deletes), VERSIONS (embedding models), and ADVERSARIES
-> so the engineering is in the interactions: recall x latency x memory x freshness x security x cost
The code here is C# and SQL. The toy HNSW-like and IVF indexes are educational — they exist so the algorithms and their trade-offs are visible, not to replace production engines. Throughout, the rule from earlier guides holds: measure on your own data (Section 12 shows how); parameter defaults and public benchmark charts rarely transfer.
A note on specifics. Vector database products evolve quickly: index types, quantization options, filter modes, parameter names, defaults, limits, and even which engines support which features change between versions (pgvector, Azure AI Search, Qdrant, Weaviate, Milvus, Elasticsearch/OpenSearch, MongoDB Atlas, SQL Server and Azure SQL vector support, Cosmos DB, and others). The algorithms and trade-offs are durable. Verify product-specific syntax, defaults, and capabilities against current documentation, and treat SQL/JSON snippets as illustrative.
1. Vector Dimensions and Similarity Metrics
What a dimension is
A vector dimension is one number in an embedding. A model that outputs "1,536-dimensional" embeddings produces a list of 1,536 floating-point values per input. The dimension count is fixed by the embedding model (and sometimes configurable — below); every vector in an index must have the same count, produced by the same model.
COMMON DIMENSIONS 384 · 512 · 768 · 1,024 · 1,536 · 3,072 (and more) — larger models and "large" variants tend to use more.
MEANING OF EACH DIMENSION None individually. The model learns a space where DISTANCE and DIRECTION encode similarity; no single coordinate means "sports" or "formal."
What dimensions cost
MEMORY PER VECTOR = dimensions x bytes per dimension
float32 (4 B): 384-d = 1.5 KiB 768-d = 3 KiB 1,536-d = 6 KiB 3,072-d = 12 KiB
float16 (2 B): half of that int8 (1 B): a quarter binary (1 bit): 1/32
AT SCALE float32 int8 binary
1 M x 768-d 2.9 GiB 0.7 GiB 0.09 GiB
1 M x 1,536-d 5.7 GiB 1.4 GiB 0.18 GiB
1 M x 3,072-d 11.4 GiB 2.9 GiB 0.36 GiB
100 M x 1,536-d 572 GiB 143 GiB 17.9 GiB
COMPUTE Every distance calculation is O(d): double the dimensions, roughly double the arithmetic per comparison (and memory bandwidth per visited vector).
INDEX Graph and list structures add overhead on top (Section 14), and high dimensions make approximate indexes work harder (below).
The curse of dimensionality (and why it matters less than the name suggests)
In very high-dimensional spaces, distances between random points concentrate: the nearest and farthest neighbors become similar in distance, so "nearest" is less discriminating and index structures prune less
effectively. In practice:
- Real embeddings are NOT random; they lie near a much lower-dimensional manifold (low "intrinsic dimensionality"), which is why ANN indexes still work well at 1,536 or 3,072 dimensions.
- But higher dimensions DO mean more memory, slower distance computations, and harder recall/speed trade-offs.
- More dimensions is not automatically better retrieval: evaluate on your task. Many applications lose little quality going from 3,072 to 1,024 or even 512 dimensions.
Reducing dimensions: Matryoshka embeddings and truncation
Some embedding models are trained with "Matryoshka" representation learning: the most important information is packed into the FIRST dimensions, so you can TRUNCATE the vector (keep the first 256, 512, or 1,024
values) and then RE-NORMALIZE it, retaining much of the quality at a fraction of the cost. Hosted APIs often expose this as a "dimensions" parameter. Truncating a model that was NOT trained this way degrades quality badly.
public static class VectorMath
{
public static float[] Normalize(float[] v)
{
float norm = TensorPrimitives.Norm(v); // L2 length
if (norm == 0) return v;
var result = new float[v.Length];
TensorPrimitives.Divide(v, norm, result); // scale to unit length
return result;
}
// Matryoshka-style reduction: keep the first `dims` values, then RE-NORMALIZE (cosine/dot assume unit vectors)
public static float[] TruncateAndNormalize(float[] v, int dims) => Normalize(v.AsSpan(0, dims).ToArray());
public static float Dot(float[] a, float[] b) => TensorPrimitives.Dot(a, b);
public static float L2(float[] a, float[] b) => TensorPrimitives.Distance(a, b);
public static float Cosine(float[] a, float[] b) => TensorPrimitives.CosineSimilarity(a, b);
}
Choose dimensions deliberately
START from the model's native size; then TEST smaller sizes (via the API's dimensions option or truncation where the model supports it) on your benchmark. If quality holds at 1,024 or 512 dimensions, you cut memory,
index size, and latency proportionally — often the cheapest scaling lever there is (Section 12).
RECORD the dimension count in the index schema and with every stored vector's embedding spec (Section 10); a dimension mismatch is a hard error in most engines, and a silent one (truncated or padded) in some.
LIMITS: engines cap dimensions per vector or per index (for example, some index types cap at a few thousand dimensions). Check the limits BEFORE choosing a model.
2. Cosine Similarity vs. Dot Product vs. Euclidean Distance
The three metrics
DOT PRODUCT (inner product) a · b = Σ a_i b_i Larger = more similar. Depends on BOTH direction and magnitude.
COSINE SIMILARITY (a · b) / (‖a‖ ‖b‖) Range -1..1. Depends on DIRECTION only (the angle); ignores magnitude.
EUCLIDEAN (L2) DISTANCE ‖a − b‖ = sqrt(Σ (a_i − b_i)²) Smaller = more similar. Depends on both direction and magnitude (straight-line distance).
(Others: Manhattan/L1 distance; Hamming distance for binary vectors; "cosine distance" = 1 − cosine similarity.)
The key fact: for unit-length vectors they are equivalent
If every vector is normalized to length 1 (many embedding models output normalized vectors; you can normalize yourself):
‖a − b‖² = ‖a‖² + ‖b‖² − 2 (a · b) = 2 − 2 (a · b) = 2 (1 − cos(a, b)) (because ‖a‖ = ‖b‖ = 1, and a · b = cos(a, b))
THEREFORE for normalized vectors:
dot product = cosine similarity
Euclidean distance² = 2 − 2 x cosine -> all three produce the IDENTICAL ranking of neighbors.
Worked example: a = [0.6, 0.8] b = [1, 0] (both unit length)
cosine = 0.6 dot = 0.6 L2² = (0.4)² + (0.8)² = 0.80 = 2 − 2 x 0.6 ✓
float[] a = VectorMath.Normalize([0.6f, 0.8f]);
float[] b = [1f, 0f];
float cosine = VectorMath.Cosine(a, b); // 0.60
float dot = VectorMath.Dot(a, b); // 0.60 — same, because both are unit length
float l2sq = MathF.Pow(VectorMath.L2(a, b), 2); // 0.80 = 2 - 2 * 0.60
Which to use — a decision guide
1. USE WHAT THE MODEL WAS TRAINED FOR. The embedding model's documentation says which metric its vectors are meant for. Most text embedding models: cosine (equivalently dot product on normalized output).
2. IF VECTORS ARE NORMALIZED -> DOT PRODUCT. It's the cheapest (no norms to compute), and ranks identically to cosine and L2. Many engines' fastest path is inner product on unit vectors.
3. IF VECTORS ARE NOT NORMALIZED:
COSINE when magnitude is an artifact (typical for text embeddings) and you want pure semantic direction.
DOT when magnitude carries meaning (for example, popularity or confidence encoded in the norm, as in some recommender models) and you WANT it to influence ranking — "maximum inner product search" (MIPS).
L2 when the space's geometry is distance-based (some image/vision embeddings; clustering) and magnitude is meaningful.
4. BINARY VECTORS -> HAMMING distance (count of differing bits), the natural metric for binary-quantized embeddings (Section 12).
5. NORMALIZE ON WRITE (and for queries) so you can switch among cosine, dot, and L2 later without re-embedding, and so every metric agrees.
Gotchas
INDEX METRIC MUST MATCH QUERY METRIC An HNSW/IVF index is built for ONE metric. Querying with another either fails, silently ignores the index (a full scan), or returns wrong neighbors.
(pgvector: the operator class in CREATE INDEX — vector_cosine_ops, vector_ip_ops, vector_l2_ops — must match the operator in ORDER BY: <=>, <#>, <->.)
DISTANCE vs. SIMILARITY Engines often return DISTANCES (smaller is better) rather than similarities. pgvector's <#> returns the NEGATIVE inner product so ORDER BY ascending works. Convert for display:
cosine similarity = 1 − cosine distance; dot similarity = −(<#> value).
MAGNITUDE SURPRISES Using dot product on UNNORMALIZED text embeddings lets long or "confident" vectors dominate rankings — usually not what you want.
MIXED SOURCES Mixing normalized and unnormalized vectors in one index breaks dot-product and L2 rankings; enforce normalization in the ingestion pipeline and assert it in tests.
SCORE THRESHOLDS A similarity cutoff ("keep > 0.80") is meaningful only for ONE model and metric; re-calibrate whenever either changes.
HIGH SIMILARITY ≠ RELEVANCE Embedding similarity is a retrieval heuristic; use rerankers and thresholds calibrated on labeled data (see Advanced RAG Architecture, Section 12).
Exact top-k (the baseline every ANN index is measured against)
// Brute-force top-k by dot product (assumes unit vectors, so this equals cosine). Exact, O(N x d) — the ground truth for measuring ANN recall.
public static List<(int Id, float Score)> ExactTopK(IReadOnlyList<float[]> vectors, float[] query, int k)
{
var heap = new PriorityQueue<int, float>(); // min-heap: the root is the WORST of the current top-k
for (int i = 0; i < vectors.Count; i++)
{
float score = TensorPrimitives.Dot(vectors[i], query);
if (heap.Count < k) heap.Enqueue(i, score);
else if (heap.TryPeek(out _, out float worst) && score > worst) heap.EnqueueDequeue(i, score); // replace the worst
}
var results = new List<(int, float)>(heap.Count);
while (heap.TryDequeue(out int id, out float score)) results.Add((id, score));
results.Reverse(); // best first
return results;
}
3. Approximate Nearest-Neighbor Search
Why approximate
Exact search compares the query to every vector: cost grows linearly with N. At 100 million vectors of 1,536 dimensions, one query touches ~570 GiB of data. Approximate nearest-neighbor (ANN) indexes organize vectors so a query examines only a small, promising fraction — returning almost the true nearest neighbors in a tiny fraction of the time. You accept imperfect recall in exchange for speed and memory savings.
Recall: the accuracy measure
RECALL@k = | ANN top-k ∩ exact top-k | / k
Recall@10 = 0.95 means the ANN index returned, on average, 9.5 of the 10 true nearest neighbors.
It is measured against EXACT search on a sample of queries — there is no other ground truth for "nearest."
Recall is a property of (index + parameters + data + filters). It must be MEASURED on your data; defaults and public benchmarks don't transfer.
public static double RecallAtK(IReadOnlyList<int> approximate, IReadOnlyList<int> exact, int k) =>
approximate.Take(k).Intersect(exact.Take(k)).Count() / (double)k;
// Average over many queries (use a few hundred representative ones; hold their exact answers as ground truth)
double MeanRecall(IEnumerable<(IReadOnlyList<int> Approx, IReadOnlyList<int> Exact)> results, int k) => results.Average(r => RecallAtK(r.Approx, r.Exact, k));
The families of ANN methods
GRAPH-BASED Build a proximity graph; search by greedy traversal toward the query. HNSW (Section 4), DiskANN/Vamana (graph on SSD + compressed vectors in RAM).
Best recall/latency trade-off in RAM; heavier build and memory; hard to update and delete cleanly.
PARTITION-BASED Divide space into cells; search only the nearest cells. IVF (Section 5), plus variants. Cheaper to build and smaller; needs training; recall tied to how many cells you probe.
QUANTIZATION-BASED Compress vectors (scalar/int8, binary, product quantization) for smaller memory and faster approximate scoring; usually combined with the above and with RESCORING.
TREE-BASED Space-partitioning trees (for example Annoy's random-projection forests). Simple and read-mostly; generally superseded by graphs for high-dimensional text embeddings.
HASH-BASED Locality-sensitive hashing (LSH). Theoretically elegant, rarely the best in practice for embeddings today.
HYBRID COMBINATIONS IVF-PQ, HNSW + scalar quantization, DiskANN + PQ — real engines mix techniques (e.g. a compressed index for candidate generation, full-precision vectors to rescore).
The trade-off triangle
RECALL (accuracy)
/\
/ \
/ \
LATENCY ------- MEMORY / COST
Every index parameter moves you along these axes: examine more candidates (higher recall, slower), store more links or full-precision vectors (higher recall, more memory), compress (less memory, lower recall unless you rescore).
There is no free setting — only the right trade-off for your latency SLO, recall target, and budget. Section 12 shows how to find it empirically.
Exact search is not the enemy
USE EXACT SEARCH WHEN
- N is small (thousands to a few hundred thousand, depending on dimensions and hardware) — a brute-force SIMD scan can be faster than maintaining an index.
- A FILTER is highly selective (only a few thousand candidates match) — score the filtered set exactly (Section 6).
- You need a perfect-recall BASELINE for evaluation, or a fallback when an ANN query returns too few results.
- Recall requirements are strict (compliance and discovery workloads).
Most engines offer an exact/"flat" mode alongside ANN indexes for exactly these cases.
4. HNSW Indexing
The idea: a small-world graph with express lanes
HNSW (Hierarchical Navigable Small World) is the dominant in-memory ANN index. It builds a graph where each vector is a node linked to nearby vectors, arranged in layers:
LAYER 2 (very sparse): o─────────────────o <- long-range "express lanes"; few nodes
LAYER 1 (sparser): o────o────o───────o────o────o
LAYER 0 (ALL vectors): o─o─o─o─o─o─o─o─o─o─o─o─o─o─o─o─o─o─o <- every vector, linked to its nearest neighbors
SEARCH: start at the entry point in the top layer ──> greedily hop to whichever neighbor is closer to the query ──> when no neighbor is closer, DROP to the next layer at that node ──>
repeat down to layer 0 ──> at layer 0 run a best-first (beam) search that keeps the `ef` best candidates, expanding the closest unvisited ones ──> return the best k.
The upper layers let the search cross the space in a few long hops; layer 0 does the precise local search. Logarithmic-ish scaling in N is why HNSW is so fast.
Building the graph
EACH NEW VECTOR
1. LEVEL: draw a random maximum layer (exponentially decaying: most vectors only live on layer 0; roughly 1 in M reaches layer 1, 1 in M² layer 2...).
2. DESCEND from the entry point to find its place, as in a search.
3. At each layer from its level down to 0: search for the efConstruction nearest candidates, then CONNECT to up to M of them (layer 0 allows up to 2M links), choosing neighbors with a DIVERSITY heuristic
(prefer neighbors that aren't already close to each other, preserving long-range links).
4. Add reverse links; if a neighbor exceeds its link limit, prune its weakest links.
Insertion is incremental — no global training step — but each insert costs a search, so BUILD is expensive and parallelism is limited by graph contention.
The parameters
M Max links per node (layer 0 holds up to 2M). Higher M = better recall, especially on hard/high-dimensional data, and denser graphs — but more memory, slower build, slower queries.
Typical 16-48.
efConstruction Candidate-list size during BUILD. Higher = a better-connected graph (higher achievable recall) and slower, costlier builds. Typical 100-400+. Set once at build time; changing it means rebuilding.
efSearch (ef) Candidate-list size at QUERY time (must be >= k). THE main runtime knob: higher = higher recall and higher latency. Typical 40-500. Tunable per query/session with no rebuild.
(pgvector: m = 16, ef_construction = 64, hnsw.ef_search = 40 are defaults; Azure AI Search documents its own defaults. Defaults are conservative starting points, not recommendations for your data.)
RULES OF THUMB
- Raise efSearch FIRST (free, instant) until recall plateaus or latency breaks your SLO.
- If recall plateaus below target even at high efSearch, the graph is the limit: rebuild with higher M and/or efConstruction.
- High-dimensional, clustered, or hard datasets need larger M; small low-dimensional ones need less.
- Recall depends on k: asking for top-100 at the efSearch tuned for top-10 hurts recall; efSearch should scale with k.
A toy navigable-graph index (to see the mechanics)
// A single-layer proximity graph with beam search — the layer-0 half of HNSW. Real HNSW adds the sparse upper layers for fast long-range navigation
// and builds incrementally in O(log N)-ish per insert; this toy builds by brute force (O(N²)) purely for illustration. Vectors must be unit-length.
public sealed class ToyNswIndex
{
private readonly IReadOnlyList<float[]> _vectors;
private readonly int[][] _neighbors;
public ToyNswIndex(IReadOnlyList<float[]> vectors, int m)
{
_vectors = vectors;
var links = new HashSet<int>[vectors.Count];
for (int i = 0; i < links.Length; i++) links[i] = [];
// Link every vector to its M nearest neighbors, in BOTH directions (bidirectional links keep the graph navigable)
Parallel.For(0, vectors.Count, i =>
{
int[] nearest = Enumerable.Range(0, vectors.Count).Where(j => j != i)
.OrderByDescending(j => TensorPrimitives.Dot(vectors[i], vectors[j])).Take(m).ToArray();
lock (links) foreach (int j in nearest) { links[i].Add(j); links[j].Add(i); }
});
_neighbors = links.Select(s => s.ToArray()).ToArray();
}
public List<(int Id, float Score)> Search(float[] query, int k, int ef, int entryPoint = 0)
{
var visited = new HashSet<int> { entryPoint };
var candidates = new PriorityQueue<int, float>(); // best-first frontier (min-heap on NEGATIVE score => best candidate pops first)
var results = new SortedSet<(float Score, int Id)>(); // the ef best found so far (ascending by score; Min = worst kept)
float entryScore = TensorPrimitives.Dot(query, _vectors[entryPoint]);
candidates.Enqueue(entryPoint, -entryScore);
results.Add((entryScore, entryPoint));
while (candidates.TryDequeue(out int current, out float negScore))
{
// Stop when the best remaining candidate is worse than everything we're keeping: no neighbor of it can improve the result set
if (results.Count >= ef && -negScore < results.Min.Score) break;
foreach (int neighbor in _neighbors[current])
{
if (!visited.Add(neighbor)) continue;
float score = TensorPrimitives.Dot(query, _vectors[neighbor]);
if (results.Count < ef || score > results.Min.Score)
{
candidates.Enqueue(neighbor, -score);
results.Add((score, neighbor));
if (results.Count > ef) results.Remove(results.Min); // keep only the ef best
}
}
}
return results.Reverse().Take(k).Select(r => (r.Id, r.Score)).ToList();
}
}
// Increase `ef` and recall rises while more vectors are scored (higher latency): the efSearch trade-off in 40 lines.
HNSW operational realities
MEMORY-HUNGRY, RAM-RESIDENT The graph and (usually) the full vectors must be in memory for fast queries: traversal is random access. If the index spills out of RAM, latency collapses ("falls off a cliff").
Plan memory with headroom (Section 14); consider quantization or disk-oriented designs (DiskANN) beyond RAM.
BUILD COST Building and rebuilding large HNSW indexes takes real time and memory; parameters like efConstruction multiply it. Build off the serving path.
INSERTS ARE EASY Incremental inserts work without retraining.
DELETES AND UPDATES ARE HARD Removing a node punches a hole in the graph and can disconnect regions. Engines use tombstones (deleted flags) and later repair or rebuild; heavy churn degrades recall and wastes memory until
compaction/reindex (Section 9). An update is usually delete + insert.
FILTERS ARE HARD A filter that rejects most nodes can leave the traversal with nowhere to go (Section 6).
RECALL IS ALWAYS < 100% Even a well-tuned index misses a few true neighbors; it is stochastic in construction (random levels), so two builds differ slightly.
CONCURRENCY Reads scale well; concurrent writes contend. Many engines use segments (see below).
SEGMENTED ENGINES Several systems build one HNSW graph PER SEGMENT and merge periodically; a query searches all segments and merges top-k. More segments = more fan-out latency; merge/compaction policy matters.
5. IVF Indexing Concepts
The idea: partition, then search only nearby partitions
IVF (Inverted File) clusters the vectors into nlist cells with k-means and stores each vector in the posting list of its nearest centroid. A query finds the nprobe nearest centroids and scans only those lists.
TRAIN: run k-means over (a sample of) the vectors -> nlist centroids (cell centers)
ADD: assign each vector to its nearest centroid -> inverted lists: centroid 17 -> [vector ids...]
QUERY: 1. compare the query to the nlist centroids (cheap: nlist << N)
2. pick the nprobe closest cells
3. scan ONLY the vectors in those cells (exactly, or approximately if compressed), return the top k
Work per query ≈ nlist + (nprobe / nlist) x N vector comparisons — a fraction of the exact scan.
The parameters
nlist Number of cells. Common guidance: roughly sqrt(N) to 4 x sqrt(N) (pgvector suggests about N/1000 for up to ~1 M rows and sqrt(N) beyond). More cells = smaller lists = faster scans per probe,
but more probes needed for the same recall and a costlier assignment step.
nprobe Cells searched per query. THE runtime knob: higher = higher recall, slower. Often starts near sqrt(nlist); at nprobe = nlist it's an exact search.
(pgvector: ivfflat.probes.)
TRAINING DATA Centroids must reflect the real distribution: train on representative data (for pgvector IVFFlat, build the index AFTER loading data). If the data drifts, the clusters get stale.
A toy IVF index
// k-means partitioning + nprobe scanning (unit vectors, dot-product similarity). Educational: real engines use k-means++ init, mini-batch training, SIMD scans, and compressed lists.
public sealed class ToyIvfIndex
{
private readonly IReadOnlyList<float[]> _vectors;
private readonly float[][] _centroids;
private readonly List<int>[] _lists;
public ToyIvfIndex(IReadOnlyList<float[]> vectors, int nlist, int iterations = 10, int seed = 1)
{
_vectors = vectors;
var rng = new Random(seed);
_centroids = Enumerable.Range(0, nlist).Select(_ => (float[])vectors[rng.Next(vectors.Count)].Clone()).ToArray(); // random init (k-means++ is better)
_lists = new List<int>[nlist];
for (int iter = 0; iter < iterations; iter++)
{
for (int c = 0; c < nlist; c++) _lists[c] = [];
for (int i = 0; i < vectors.Count; i++) _lists[NearestCentroid(vectors[i])].Add(i); // assign every vector to its closest centroid
for (int c = 0; c < nlist; c++) // move each centroid to the (re-normalized) mean of its members
{
if (_lists[c].Count == 0) continue;
var mean = new float[vectors[0].Length];
foreach (int id in _lists[c]) TensorPrimitives.Add(mean, vectors[id], mean);
_centroids[c] = VectorMath.Normalize(mean);
}
}
for (int c = 0; c < nlist; c++) _lists[c] = [];
for (int i = 0; i < vectors.Count; i++) _lists[NearestCentroid(vectors[i])].Add(i); // final assignment with the converged centroids
}
private int NearestCentroid(float[] v)
{
int best = 0; float bestScore = float.NegativeInfinity;
for (int c = 0; c < _centroids.Length; c++)
{
float s = TensorPrimitives.Dot(v, _centroids[c]);
if (s > bestScore) { bestScore = s; best = c; }
}
return best;
}
public List<(int Id, float Score)> Search(float[] query, int k, int nprobe)
{
// The nprobe cells whose centroids are closest to the query...
IEnumerable<int> cells = Enumerable.Range(0, _centroids.Length).OrderByDescending(c => TensorPrimitives.Dot(query, _centroids[c])).Take(nprobe);
// ...scanned exhaustively; everything outside them is never looked at (that's both the speedup and the source of missed neighbors)
return cells.SelectMany(c => _lists[c])
.Select(id => (Id: id, Score: TensorPrimitives.Dot(query, _vectors[id])))
.OrderByDescending(x => x.Score).Take(k).ToList();
}
}
Quantization: shrinking vectors (IVF-PQ and friends)
SCALAR QUANTIZATION (SQ) Map each float32 dimension to int8 (or int4): 4x (or 8x) smaller; fast integer math; small recall loss; widely supported.
BINARY QUANTIZATION (BQ) Keep only the SIGN of each dimension (1 bit): 32x smaller; Hamming distance via XOR + popcount is extremely fast; larger recall loss — works best on high-dimensional embeddings and
when followed by rescoring (Section 12).
PRODUCT QUANTIZATION (PQ) Split the vector into m sub-vectors; learn a 256-entry codebook per sub-vector (k-means); store each sub-vector as a 1-byte code. A 1,536-d float32 vector (6,144 B) with m = 192
becomes 192 B — 32x smaller. Distances are computed from precomputed lookup tables ("asymmetric distance computation"). Higher compression, higher error than SQ.
RESCORING (RERANKING) The standard recovery: use the compressed vectors to fetch an OVERSAMPLED shortlist (say 3-10x k), then re-score those few candidates with the full-precision vectors (kept on disk or in
slower storage) and return the best k. Recovers most of the lost recall for a modest cost.
IVF versus HNSW
| IVF (Flat / SQ / PQ) | HNSW | |
|---|---|---|
| Recall/latency (in RAM) | Good; usually behind HNSW at equal recall | Excellent — the usual leader |
| Build time | Fast (k-means + assignment) | Slow (graph construction) |
| Memory | Small, especially with PQ/SQ | Larger (links + full vectors in RAM) |
| Training | Required; representative data matters; drift hurts | None (incremental) |
| Inserts | Cheap (append to nearest list) but centroids go stale | Supported, cost per insert is a search |
| Deletes | Easy (remove from the list) | Hard (tombstones, repair, rebuild) |
| Filtering | Natural: skip non-matching vectors while scanning lists | Hard: the graph may disconnect |
| Scaling beyond RAM | Natural with compressed lists and disk-resident full vectors | Needs disk-oriented variants (DiskANN-style) |
| Main tuning knobs | nlist (build), nprobe (query), PQ settings | M, efConstruction (build), efSearch (query) |
| Best for | Very large or memory-constrained datasets, frequently rebuilt/changing data, cost-sensitive scale | Latency-critical, high-recall search on data that fits in memory |
CHOOSING: start with what your engine makes easy and measure. HNSW is the default for most RAG-scale workloads (up to tens of millions of vectors in RAM). IVF-PQ-style or disk-oriented graph indexes enter when memory
cost dominates (hundreds of millions to billions of vectors). Many engines default to HNSW and add quantization as the memory lever.
6. Vector Indexing and Filtering
Filters are where approximate search gets difficult
Real queries almost always carry constraints — tenant, permissions, document type, date, language — and the Advanced RAG guide stressed that security filters are mandatory. But filtering and approximate indexes interact badly: an ANN index is built over all vectors, while the filter selects an arbitrary subset at query time.
The three basic strategies
POST-FILTER Run ANN to get the top k' candidates, THEN drop those that fail the filter.
+ Simple; works with any index.
- If the filter is selective, most candidates are discarded and you return FEWER than k results (possibly zero) even though matches exist. Over-fetching (k' >> k) helps, with diminishing returns.
PRE-FILTER Compute the set of matching vectors first, then search only within it.
+ Always returns k results if k match; correct for selective filters.
- Without index support, "search within a subset" means a brute-force scan of the subset — fine when it is small, slow when it is large.
FILTER-DURING-SEARCH The index traversal itself checks the predicate (graph nodes or list entries that fail are skipped, but the walk continues through them to stay connected).
+ The best of both when implemented well. Engines differ widely in quality and in recall under selective filters.
- May need larger candidate lists (ef, probes) as the filter gets more selective; some implementations degrade badly at very selective filters.
Selectivity decides everything
SELECTIVITY = the fraction of vectors that pass the filter.
~100% (broad filter) The filter barely matters; ordinary ANN works.
10-50% Filter-during-search or post-filter with moderate over-fetch works; verify recall.
1-10% Danger zone for post-filtering; in-graph filtering needs wider search; consider specialized support (payload-aware graph links, partitions).
< ~1%, or a few thousand matches BEST DONE EXACTLY: filter first (an ordinary index on the filter field), then score the few matching vectors by brute force — perfect recall, and cheap because the set is small.
Tenant filters are the classic highly selective case: one tenant among thousands is a fraction of a percent of the data — so naive ANN + tenant filter gives poor recall for small tenants (Section 8).
A selectivity-aware search planner
// Choose the execution strategy per query from the filter's estimated selectivity. Thresholds are starting points — calibrate per engine on your benchmark.
public sealed class FilteredSearchPlanner(IVectorStats stats, IAnnIndex index, IExactScanner exact, int exactMaxCandidates = 20_000)
{
public async Task<IReadOnlyList<Hit>> SearchAsync(float[] query, VectorFilter filter, int k, CancellationToken ct)
{
long matching = await stats.EstimateMatchesAsync(filter, ct); // from a metadata index / statistics, not a scan
double selectivity = matching / (double)await stats.TotalCountAsync(ct);
// 1. Very selective: score the few matching vectors exactly — perfect recall, bounded cost
if (matching <= exactMaxCandidates)
return await exact.ScanAsync(query, filter, k, ct);
// 2. Broad filter: ordinary ANN; the filter hardly changes the neighborhood structure
if (selectivity >= 0.5)
return await index.SearchAsync(query, k, filter, new AnnOptions(EfSearchMultiplier: 1.0, Oversample: 1.0), ct);
// 3. In between: filter during traversal with a wider search and some over-fetch so filtering can't starve the result set
return await index.SearchAsync(query, k, filter, new AnnOptions(EfSearchMultiplier: 2.0, Oversample: 3.0), ct);
}
}
Making filters fast and correct
INDEX THE FILTER FIELDS Declare filterable/indexed payload fields (a B-tree or inverted index on tenant_id, doc_type, date...). An unindexed filter forces a scan of every record's metadata.
(Azure AI Search "filterable"; MongoDB Atlas "filter" fields in the vector index definition; Qdrant payload indexes; PostgreSQL indexes on the columns.)
KEEP FILTERS SIMPLE Equality and small IN lists on indexed fields are cheap; complex expressions, regex, and functions on fields are not.
PARTITION BY THE HOT FILTER If nearly every query filters on one field (tenant), make it a partition or a separate index (Section 8) rather than a filter.
VERIFY FILTERED RECALL Measure recall at several selectivities (100%, 50%, 10%, 1%) against exact filtered search. Unfiltered recall tells you nothing about filtered recall.
RETURN K OR SAY WHY If fewer than k results come back, you can't tell "only 4 matches exist" from "the index starved" — log selectivity and result counts; fall back to exact scan when the count is suspiciously low.
SECURITY FILTERS ARE NOT OPTIONAL Never rely on post-filtering "usually good enough" for access control: derive them from identity and apply them inside the query (see AI Security, Section 8).
PostgreSQL / pgvector: concrete patterns
-- HNSW built for cosine distance, matching the <=> operator used in queries
CREATE INDEX chunks_embedding_hnsw ON chunks USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 128);
-- A filter that would starve a plain HNSW scan: pgvector 0.8+ can keep scanning the index until enough rows pass the filter
SET hnsw.iterative_scan = relaxed_order; -- strict_order preserves exact ordering at more cost; relaxed_order is faster and slightly looser
SET hnsw.max_scan_tuples = 20000; -- cap the work spent hunting for matches
SET hnsw.ef_search = 100;
SELECT id, content
FROM chunks
WHERE tenant_id = 42 AND doc_type = 'policy'
ORDER BY embedding <=> $1 -- the operator must match the index's operator class, and LIMIT is required for index use
LIMIT 10;
-- Very selective filter + a normal B-tree index on the filter column: the planner can filter first and sort the few survivors exactly
CREATE INDEX chunks_tenant_idx ON chunks (tenant_id, doc_type);
-- A large tenant gets its own partial index (the query's WHERE tenant_id = 42 must match the index predicate)
CREATE INDEX chunks_t42_hnsw ON chunks USING hnsw (embedding vector_cosine_ops) WHERE tenant_id = 42;
-- Many tenants: partition the table so each partition has its own (smaller, better-recall) HNSW index
CREATE TABLE chunks (id bigint, tenant_id int, embedding vector(1536), content text) PARTITION BY HASH (tenant_id);
7. Hybrid Indexes
Several index structures behind one query
Advanced RAG Architecture, Sections 10–11, covered hybrid retrieval (keyword + vector, fused by RRF). This section covers the index engineering underneath.
A HYBRID SYSTEM CONTAINS
INVERTED INDEX (BM25) term -> posting list of documents; exact and phrase matching; language analyzers, stemming, synonyms.
ANN INDEX HNSW/IVF over dense embeddings.
METADATA INDEXES B-tree/inverted indexes for filters, shared by both legs.
(OPTIONAL) SPARSE-VECTOR INDEX learned sparse representations (SPLADE-style) stored as sparse vectors.
(OPTIONAL) MULTI-VECTOR INDEX several vectors per record (see below).
One engine or two?
ONE ENGINE (Azure AI Search, Elasticsearch/OpenSearch, Weaviate, Qdrant with sparse vectors, PostgreSQL with FTS + pgvector, MongoDB Atlas Search + Vector Search)
+ One filter definition applies to both legs; one security model; one write path (no dual-write inconsistency); simpler operations; some engines fuse server-side (RRF) in a single call.
- You inherit that engine's strengths and weaknesses on each leg; fusion options may be fixed.
TWO ENGINES (a search engine for text + a vector database for embeddings, fused in your application)
+ Best-of-breed per leg; independent scaling.
- Dual writes and EVENTUAL CONSISTENCY between legs (a document visible in one index and not the other); filters and ACLs defined twice (drift = a security bug); more operations; fusion code to own.
Index-level considerations
CONSISTENT FRESHNESS Keyword and vector segments may refresh at different intervals; a freshly updated document can appear in one leg first. Measure and bound the skew.
SHARED FILTERS The SAME tenant/ACL predicate must apply to both legs, enforced in one place.
ANALYZERS Per-language analyzers, synonyms, n-gram fields for codes/IDs, keyword (non-analyzed) fields for exact identifiers — and fuzzy matching to tolerate OCR errors.
FIELD STRATEGY Searchable text fields (title, body) with different boosts; separate vector fields for different content (title vs. body, or different models); filterable metadata fields.
SCORE FUSION Prefer rank-based fusion (RRF) so legs' incomparable scores don't matter; if you blend scores, normalize per query (min-max or z-score) and tune the weight on a benchmark.
INDEX SIZE The inverted index is usually far smaller than the vector index; vectors dominate memory and cost.
Sparse vectors
DENSE EMBEDDINGS Every dimension has a value (e.g. 1,536 floats). Capture semantic similarity.
SPARSE VECTORS Mostly zeros; stored as (term id -> weight) pairs. BM25 and learned sparse models (SPLADE, ELSER-style) produce them: keyword-like exactness plus learned term expansion. Some vector engines store
sparse and dense vectors in the SAME record and fuse server-side (sparse-dense hybrid).
// Sparse dot product over (termId -> weight) dictionaries: iterate the SMALLER vector and look up in the larger one
public static float SparseDot(Dictionary<int, float> a, Dictionary<int, float> b)
{
if (a.Count > b.Count) (a, b) = (b, a);
float sum = 0;
foreach (var (term, weight) in a)
if (b.TryGetValue(term, out float other)) sum += weight * other;
return sum;
}
// Weighted dense + sparse blend with per-query min-max normalization (use RRF instead when you don't want to tune alpha)
public static IReadOnlyList<(string Id, double Score)> Blend(IReadOnlyDictionary<string, float> dense, IReadOnlyDictionary<string, float> sparse, double alpha)
{
static Dictionary<string, double> MinMax(IReadOnlyDictionary<string, float> scores)
{
if (scores.Count == 0) return [];
float min = scores.Values.Min(), max = scores.Values.Max();
return scores.ToDictionary(kv => kv.Key, kv => max == min ? 1.0 : (kv.Value - min) / (double)(max - min));
}
var d = MinMax(dense); var s = MinMax(sparse);
return d.Keys.Union(s.Keys)
.Select(id => (Id: id, Score: alpha * d.GetValueOrDefault(id) + (1 - alpha) * s.GetValueOrDefault(id)))
.OrderByDescending(x => x.Score).ToList();
}
Multi-vector and late interaction
MULTIPLE VECTORS PER RECORD Several embeddings point to one record: chunk text, an LLM summary, hypothetical questions (see Advanced RAG Architecture, Section 6) — "named vectors" or multiple vector fields.
LATE INTERACTION (ColBERT-style) Store one vector PER TOKEN; score a query-document pair with MaxSim (for each query token, take its best-matching document token, sum). Much higher accuracy on some tasks, much larger
storage (hundreds of vectors per document) and heavier queries; supported as "multivectors" in some engines. Usually used for reranking a shortlist rather than as the first-stage index.
MULTIPLE MODELS Separate vector fields for different embedding models (also the foundation of zero-downtime model migration, Section 10).
8. Multi-Tenancy
Many customers, one platform: choose the isolation model deliberately
In a multi-tenant system, tenants share infrastructure but must never see each other's data, should not degrade each other's performance, and often need separate lifecycle (export, deletion), keys, and sometimes models. The vector database layout is the foundation of all of that.
The four models
1. POOLED (shared index + tenant_id filter)
All tenants' vectors in one index; every query filters on tenant_id.
+ Cheapest, simplest to operate, scales to very many small tenants, one index to tune and monitor.
- Weakest isolation: one missing filter = a cross-tenant leak. Noisy neighbors share resources. Small tenants suffer FILTERED-ANN recall problems (Section 6). Per-tenant deletion/export are filter-and-scan operations.
2. SILO (an index/collection per tenant)
+ Strongest logical isolation; per-tenant tuning, models, and keys; trivial export and deletion (drop the index); no filtered-recall problem; noisy neighbors contained.
- Per-index overhead (memory, file handles, metadata); engine limits on index/collection counts (Azure AI Search tiers cap indexes; others degrade with thousands of collections); cold tenants still cost something;
operational complexity (provisioning, migrations across N indexes).
3. PARTITIONED / NAMESPACED (partitions or namespaces inside one index)
A middle path: each tenant is a partition, namespace, shard, or "tenant" object inside a shared deployment — e.g. Pinecone namespaces, Weaviate multi-tenancy (per-tenant shards, hot/cold offloading), Qdrant payload-based
multi-tenancy with tenant-aware indexing, Milvus partition keys, PostgreSQL declarative partitioning or partial indexes.
+ Near-silo query performance and recall (each partition is searched on its own), cheap tenant deletion (drop the partition), shared infrastructure. Mind per-engine limits on partition counts.
- Engine-specific; some engines still share resources across partitions.
4. DEPLOYMENT PER TENANT (database/cluster/subscription per tenant)
+ Maximum isolation, dedicated keys and networking, regulatory comfort, independent scaling and upgrades.
- Highest cost and operational burden — reserved for large or regulated customers.
BRIDGE (TIERED) MODEL: pool the long tail of small tenants; give large, regulated, or premium tenants their own partition, index, or deployment. Automate tenant placement and migration between tiers.
Choosing
| Factor | Pooled | Partitioned | Silo (index per tenant) | Deployment per tenant |
|---|---|---|---|---|
| Number of tenants | Thousands to millions | Hundreds to many thousands (engine limits) | Tens to hundreds | A handful |
| Isolation strength | Logical (filter) | Logical, stronger | Strong | Strongest |
| Filtered-recall risk | High for small tenants | Low | None | None |
| Per-tenant tuning / model / keys | Hard | Limited | Easy | Easiest |
| Deletion / export | Filter + scan | Drop partition | Drop index | Drop deployment |
| Cost per tenant | Lowest | Low | Medium | Highest |
| Operational overhead | Lowest | Low–medium | Medium–high | Highest |
Enforce isolation where it cannot be bypassed
NEVER trust callers to remember the tenant filter. Put isolation in a layer that every query must pass through:
- A tenant-scoped REPOSITORY / data-access class that takes the tenant from the authenticated principal (never a request parameter or a model-supplied argument) and ALWAYS injects the filter.
- DATABASE-LEVEL enforcement where available: PostgreSQL row-level security (below), per-tenant schemas/roles, or per-tenant API keys scoped to one index/namespace.
- TESTS: automated cross-tenant leakage tests in CI (tenant A must never retrieve tenant B's documents) — against the real engine, with adversarial filter inputs (see AI Security, Section 14).
-- PostgreSQL row-level security: even a buggy query cannot read another tenant's rows
ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE chunks FORCE ROW LEVEL SECURITY; -- apply to the table owner as well (superusers and roles with BYPASSRLS still bypass: never use those for the app)
CREATE POLICY tenant_isolation ON chunks
USING (tenant_id = current_setting('app.tenant_id', true)::int); -- missing setting => NULL => no rows (fails CLOSED)
// Set the tenant for ONE TRANSACTION (set_config(..., true) = local to the transaction). Never set it at session level on a pooled connection —
// the setting would persist on the connection and leak into the NEXT request that borrows it.
public async Task<IReadOnlyList<Hit>> SearchAsync(int tenantId, float[] queryVector, int k, CancellationToken ct)
{
await using NpgsqlConnection conn = await dataSource.OpenConnectionAsync(ct);
await using NpgsqlTransaction tx = await conn.BeginTransactionAsync(ct);
await using (var setTenant = new NpgsqlCommand("SELECT set_config('app.tenant_id', @t, true)", conn, tx))
{
setTenant.Parameters.AddWithValue("t", tenantId.ToString());
await setTenant.ExecuteNonQueryAsync(ct);
}
await using var search = new NpgsqlCommand("SELECT id, 1 - (embedding <=> $1) AS score FROM chunks ORDER BY embedding <=> $1 LIMIT $2", conn, tx);
search.Parameters.AddWithValue(new Pgvector.Vector(queryVector));
search.Parameters.AddWithValue(k);
var hits = new List<Hit>();
await using var reader = await search.ExecuteReaderAsync(ct);
while (await reader.ReadAsync(ct)) hits.Add(new Hit(reader.GetInt64(0).ToString(), reader.GetDouble(1)));
await tx.CommitAsync(ct);
return hits; // RLS silently added the tenant predicate; the application code never has to remember it
}
Multi-tenant concerns beyond isolation
NOISY NEIGHBORS Per-tenant rate limits, concurrency caps, max k and max filter complexity, and storage quotas (see AI Security, Section 11); isolate heavy tenants into their own partition or index.
SMALL-TENANT RECALL In pooled indexes, route small tenants to exact search or per-tenant partial indexes so a tiny tenant doesn't get a starved ANN traversal.
LIFECYCLE Onboarding (provision partition/index, quotas, keys), offboarding (verified deletion across vectors, payloads, caches, backups per policy), export on request, data-residency placement.
PER-TENANT MODELS Tenants that use different embedding models or dimensions need separate vector fields or indexes — a pooled index with one vector field can't mix them.
PER-TENANT KEYS Customer-managed encryption keys generally imply silo or deployment-per-tenant models.
COST ATTRIBUTION Track vectors stored, queries, and embedding spend per tenant for chargeback and anomaly detection.
OBSERVABILITY Tenant-level latency, error rate, recall canaries, and storage — with tenant ID as a log/trace attribute (but NOT a high-cardinality metric tag; see AI Observability, Section 3).
MIGRATION BETWEEN TIERS Moving a tenant from pooled to silo (as it grows or becomes regulated) should be an automated, verified, zero-downtime operation (Section 11).
9. Vector Data Lifecycle
A vector is derived data with a lifecycle
Each vector exists because of a source document, a chunking decision, and an embedding model — and each of those changes. Treat the vector index as derived, rebuildable data with a managed lifecycle, not as a primary store.
SOURCE -> PARSE/CHUNK -> EMBED -> INDEX -> SERVE -> UPDATE -> EXPIRE / DELETE -> ARCHIVE / PURGE
(system of record) (derived) (churn) (compliance)
What to store with every vector record
id deterministic: doc id + content hash + position (idempotent upserts)
vector (and the quantized form, if the engine holds one)
source_ref document id, source system, URL/path, page/section — to trace back and to delete by source
source_version monotonic version/timestamp of the source document — to ignore stale or out-of-order updates
content_hash detects "unchanged" so you don't re-embed
embedding_spec provider, model, model version, dimensions, normalization, preprocessing (Section 10)
chunker_version how it was produced
tenant_id, acl isolation and access control
created_at, updated_at, expires_at lifecycle and retention
deleted (tombstone) for soft deletes and out-of-order protection, until purged
metadata filterable attributes (type, language, date, product...)
Keeping the index in sync with the source
CHANGE CAPTURE Source changes reach the indexer via change feed/CDC, an OUTBOX table (transactionally written with the source change), events, or periodic diffing — NOT by dual-writing from application code
(partial failure leaves the index and source inconsistent).
IDEMPOTENT Applying the same change twice must be harmless. Deterministic IDs + upsert semantics.
ORDER-SAFE Events can arrive out of order or be replayed. Compare source_version: ignore anything not newer than what's indexed. Keep TOMBSTONES (with their version) for a retention window so a late stale
upsert can't resurrect a deleted document.
ATOMIC SWAP On a content change, write the new version's chunks, THEN delete the old version's (no gap in search).
RECONCILIATION Eventual consistency drifts. A scheduled job diffs source IDs/versions against the index and repairs missing, stale, and orphaned records; report drift as a metric.
BACKPRESSURE Bounded queues and rate-limited embedding calls; bulk loads shouldn't starve interactive queries (or be starved by them).
public enum ChangeKind { Upserted, Deleted }
public record SourceChange(string DocId, long SourceVersion, ChangeKind Kind, string? Content, int TenantId);
public sealed class IndexSynchronizer(IVectorIndex index, IChunkEmbedder embedder, ILogger<IndexSynchronizer> log)
{
public async Task ApplyAsync(SourceChange change, CancellationToken ct)
{
DocState? current = await index.GetDocStateAsync(change.DocId, ct); // latest indexed version, or a tombstone
// Duplicate or out-of-order event: the index already reflects this version or a newer one -> no-op (idempotent and order-safe)
if (current is not null && change.SourceVersion <= current.SourceVersion)
{
log.LogDebug("Ignoring stale change for {Doc}: v{Incoming} <= v{Current}", change.DocId, change.SourceVersion, current.SourceVersion);
return;
}
if (change.Kind == ChangeKind.Deleted)
{
await index.DeleteDocAsync(change.DocId, tombstoneVersion: change.SourceVersion, ct); // remove vectors; keep a versioned tombstone for a while
return;
}
string hash = Hash(change.Content!);
if (current?.ContentHash == hash)
{
await index.UpdateSourceVersionAsync(change.DocId, change.SourceVersion, ct); // metadata-only change: do NOT re-embed
return;
}
IReadOnlyList<IndexRecord> fresh = await embedder.EmbedDocumentAsync(change, hash, ct); // chunk + embed with the CURRENT embedding spec
await index.UpsertAsync(fresh, ct); // 1. write the new version...
await index.DeleteSupersededAsync(change.DocId, keepContentHash: hash, ct); // 2. ...then remove the old one
}
private static string Hash(string s) => Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(s)));
}
Updates, deletes, and index health
METADATA-ONLY UPDATE Change payload/ACL fields in place (partial-update APIs) without touching the vector or re-embedding. ACL changes must propagate quickly (security).
CONTENT UPDATE New chunks, new vectors; delete the old ones. (Re-embedding costs tokens — the content hash avoids needless work.)
DELETES IN THE INDEX ANN structures dislike deletion (Section 4). Engines mark vectors deleted (tombstones) and exclude them from results, but the graph/lists still carry them until COMPACTION, MERGE, VACUUM, or REINDEX.
Heavy churn therefore causes: wasted memory, slower queries, and degraded recall. Monitor the deleted-to-live ratio and schedule maintenance.
MAINTENANCE PostgreSQL: VACUUM and REINDEX INDEX CONCURRENTLY after heavy churn (HNSW indexes bloat). Segment-based engines: tune merge/optimizer policies. Others: scheduled compaction/rebuild into a fresh index and an
alias swap. Plan rebuilds as routine, off-peak, resource-budgeted operations.
HARD vs. SOFT DELETE Soft delete (flag + filter) is fast and reversible but leaves data stored — NOT sufficient for erasure obligations. Hard delete must actually purge vectors, payloads, caches, and (per retention policy) backups.
Retention, expiry, and erasure
RETENTION POLICIES Time-to-live per data class (chat memories 90 days, tickets 2 years). Implement as an expires_at field and a scheduled job that deletes expired records (and verifies the deletion).
RIGHT TO ERASURE A deletion request (GDPR and similar) must reach EVERY copy: source, chunks and vectors, payloads, search caches, semantic caches, conversation memories, logs/traces/captured content, evaluation sets,
fine-tuning data, derived summaries and graph nodes, replicas, and backups within your documented retention window. Embeddings derived from personal data are personal data.
DELETE BY SOURCE Index a source_ref/doc_id on every record so "delete everything derived from this document/user/tenant" is a single filtered delete — not a scan.
PROOF Log deletions (what, when, by whom, scope) and verify with a follow-up query that nothing remains.
Durability, backup, and disaster recovery
REBUILDABLE ≠ FREE TO REBUILD You CAN regenerate vectors from source documents — but re-embedding 100 M chunks takes days and real money, and re-parsing/OCR is slower still. So:
STORE EMBEDDINGS DURABLY Keep vectors (with their embedding spec) in cheap object storage (for example Parquet files in blob storage) as an "embedding lake." A rebuild or migration then needs no embedding API calls.
SNAPSHOTS Use the engine's snapshot/backup features; test RESTORES (an untested backup isn't one); keep snapshots consistent with the source-version watermark so replay resumes where the snapshot ended.
REPLICATION Replicas for availability and read scale; cross-region replicas or a warm standby for DR; know the RPO/RTO.
REBUILD RUNBOOK A documented, rehearsed procedure and timing for rebuilding from the embedding lake.
Tiering
HOT Recent/frequently queried data: full-precision vectors in RAM with HNSW.
WARM Older data: quantized vectors, or on-disk/DiskANN-style indexes, fewer replicas.
COLD Rarely queried: archived embeddings in object storage; restored or searched with slower batch tools on demand.
Move data by age or access frequency; tell users which tiers a query covered. Tiering is often the biggest cost lever at large scale.
10. Embedding Versioning
Vectors are only comparable within one embedding space
A vector means something only relative to the model that produced it. Vectors from different models — or different versions of the "same" model, or the same model with different dimensions, normalization, or text preprocessing — live in different spaces. Comparing them returns numbers, never errors, and the numbers are meaningless. So the embedding configuration is part of your data's identity.
What constitutes a "version"
MODEL AND VERSION provider + model + model version/revision (a provider may update a model behind a stable name; pin the version where possible)
DIMENSIONS including truncation (Matryoshka) settings
NORMALIZATION normalized or raw; metric assumed
TEXT PREPROCESSING what text is embedded: title/section-path prefixes, contextual enrichment, lowercasing, language handling, truncation limits
CHUNKING chunk size/strategy (changes which text each vector represents)
QUERY/DOCUMENT MODE instruction prefixes or query-vs-document encoders some models require
ANY CHANGE ABOVE = A NEW EMBEDDING SPEC = A NEW VECTOR SPACE
Make the spec explicit in code and data
public sealed record EmbeddingSpec(string Provider, string Model, string ModelVersion, int Dimensions, bool Normalized, string Preprocessing)
{
// A stable identifier stored with every vector and in the index metadata
public string Id => $"{Provider}:{Model}:{ModelVersion}:{Dimensions}:{(Normalized ? "norm" : "raw")}:{Preprocessing}";
}
public record IndexInfo(string PhysicalName, EmbeddingSpec Spec, DateTimeOffset BuiltAt);
public interface IVectorIndexRegistry
{
IndexInfo Resolve(string alias); // "docs-prod" -> the physical index currently serving, with the spec it was built from
Task SwapAliasAsync(string alias, string newPhysicalName, CancellationToken ct); // atomic cutover (and instant rollback)
}
// QUERY SIDE: embed the query with the spec of the index you are about to search — never with "whatever the default is now"
public async Task<IReadOnlyList<Hit>> SearchAsync(string alias, string question, SearchFilter filter, CancellationToken ct)
{
IndexInfo info = registry.Resolve(alias);
IEmbeddingGenerator<string, Embedding<float>> embedder = embedders.For(info.Spec); // throws if that spec isn't available to this service
float[] queryVector = (await embedder.GenerateVectorAsync(question, cancellationToken: ct)).ToArray();
if (queryVector.Length != info.Spec.Dimensions)
throw new InvalidOperationException($"Query vector has {queryVector.Length} dimensions; index {info.PhysicalName} expects {info.Spec.Dimensions}.");
return await index.SearchAsync(info.PhysicalName, queryVector, filter, ct);
}
Why you'll need to change embeddings
DEPRECATION The provider retires the model; you must move.
QUALITY A better/cheaper model, a multilingual model, or a domain-tuned one.
COST / LATENCY Smaller dimensions or a cheaper model.
CHUNKING OR TEXT CHANGES Better chunking, contextual enrichment, or new metadata prefixes change what's embedded.
FINE-TUNING You fine-tune an embedding model on your domain.
COMPLIANCE Move to a model you can host in-region or privately.
Migration strategies
1. BLUE/GREEN INDEX (the default)
Build a NEW index with the new spec alongside the live one; backfill from source or the embedding lake; evaluate against the benchmark (Advanced RAG Architecture, Section 19); shadow traffic; switch the alias;
keep the old index for a rollback window; then delete it. Costs temporary double storage and a full re-embedding — but is clean, testable, and reversible.
2. NAMED VECTORS ON THE SAME RECORDS
Add a second vector field (vector_v2) to existing records (engines with named vectors / multiple vector fields: Qdrant, Azure AI Search, Weaviate, and others). Backfill vector_v2 progressively; queries pick the field
by version; flip when complete; drop vector_v1. Avoids duplicating payloads, but index size grows during the overlap and every record must be touched.
3. DUAL-WRITE DURING MIGRATION
New and changed documents are embedded with BOTH specs while the backfill proceeds; at cutover the new index is already current.
4. PARTITION-BY-VERSION (use with care)
Records carry their spec; queries search EACH version's partition with the matching query embedding and fuse by RANK (RRF — scores from different spaces are not comparable). Workable as a temporary bridge, but doubles query
cost and complexity. Never put different specs in ONE ANN search.
5. LAZY RE-EMBEDDING ON ACCESS — generally avoid for production; mixed states are hard to reason about, and unqueried data stays on the old space forever.
Backfill planning
COST tokens to embed x price per token; estimate from the registry (document count, average tokens). Reuse unchanged-content skipping; reuse parsed/chunked artifacts (don't re-parse or re-OCR).
THROUGHPUT Bounded by provider rate limits (tokens/minute) — plan days for hundreds of millions of chunks; use batch endpoints where offered; parallelize within quotas; checkpoint progress.
PRIORITY Embed hot/high-traffic documents first so partial migrations still help.
VALIDATION Compare old vs. new on the benchmark: recall, nDCG, answer quality, latency, and cost; check filtered recall too. Don't assume the "better" model is better on YOUR data.
ROLLBACK Keep the old index and embedder available until the new one is proven in production.
CALIBRATION Similarity thresholds and reranker cutoffs are model-specific — recalibrate on the new spec before cutover.
RECORD Update the registry (which index serves which spec), and stamp traces with the spec ID (see AI Observability, Section 6).
11. Vector Database Migrations
Kinds of migration
INDEX / SCHEMA CHANGE New HNSW parameters, a different metric, quantization on/off, added filterable fields, different dimensions. -> usually a REBUILD into a new index.
ENGINE MIGRATION pgvector -> Azure AI Search, Qdrant -> Milvus, a managed service -> self-hosted, or vice versa. -> export, transform, import, verify, cut over.
EMBEDDING-MODEL MIGRATION Section 10.
VERSION UPGRADE A new engine/extension version (index format changes, new defaults, behavior changes). -> test on a copy; some upgrades require REINDEX.
TOPOLOGY CHANGE Resharding, adding replicas, moving regions, moving tenants between tiers. -> data movement with live traffic.
The core rule: move vectors, don't recompute them
If the embedding spec isn't changing, copy the EXISTING vectors and payloads. Re-embedding is slow, costly, and can subtly change results (and some embedding APIs are not perfectly reproducible). Export from the source engine
(or from your embedding lake) and import into the target.
A zero-downtime pattern
1. PREPARE Create the target (index/collection/schema) with the new settings. Define the field mapping and metric; double-check the metric and normalization match what the vectors were built for.
2. DUAL-WRITE Start applying ALL ongoing changes to BOTH systems (via the change feed/outbox — Section 9 — not by application code), so the target stays current during the backfill.
3. BACKFILL Copy historical data in batches (below), checkpointed and resumable, rate-limited so production isn't hurt.
4. VERIFY Counts per tenant/type, checksums or sampled vector equality, filter parity, and — most important — SEARCH QUALITY: recall vs. exact on benchmark queries, plus overlap with the old system's results.
5. SHADOW READS Send a copy of real queries to the target, discard its responses, and compare: result overlap@k, latency percentiles, errors, filtered-query behavior.
6. CUT OVER Switch reads gradually (feature flag / alias / percentage rollout: 1% -> 10% -> 50% -> 100%) with automatic rollback on SLO or quality regression.
7. SETTLE Keep dual-writing and the old system for a rollback window; then stop writes to the old system and decommission it after the retention period.
A resumable backfill
public sealed class VectorMigrator(IVectorSource source, IVectorTarget target, ICheckpointStore checkpoints, ILogger<VectorMigrator> log)
{
public async Task<MigrationReport> RunAsync(CancellationToken ct)
{
string? cursor = await checkpoints.LoadAsync(ct); // resume after a crash
long copied = 0;
while (true)
{
SourcePage page = await source.ReadPageAsync(cursor, pageSize: 1_000, ct);
if (page.Items.Count == 0) break;
// Schema mapping only: rename fields, convert types, translate filters. Vectors are copied AS-IS — never re-embedded.
List<TargetRecord> records = page.Items.Select(Transform).ToList();
await RetryAsync(() => target.UpsertBatchAsync(records, ct), ct); // idempotent upserts by deterministic ID: re-running a batch is harmless
copied += records.Count;
cursor = page.NextCursor;
await checkpoints.SaveAsync(cursor, ct); // checkpoint AFTER the batch is durably written
if (copied % 100_000 == 0) log.LogInformation("Migrated {Count} records", copied);
}
return await VerifyAsync(copied, ct);
}
}
// Verification must compare SEARCH BEHAVIOR, not just row counts
public static double OverlapAtK(IReadOnlyList<string> oldResults, IReadOnlyList<string> newResults, int k) =>
oldResults.Take(k).Intersect(newResults.Take(k)).Count() / (double)k;
// Shadow comparison across a sample of real queries: report the distribution of overlap@10 (and latency), and investigate queries with low overlap.
Bulk-loading performance
LOAD FIRST, INDEX AFTER Building an HNSW/IVF index incrementally during a bulk import is far slower than loading data and building the index once. (pgvector: COPY the rows, THEN CREATE INDEX — with a large
maintenance_work_mem and parallel maintenance workers; IVFFlat in particular must be built after the data is loaded so its centroids are representative. Many engines can defer
or batch indexing during imports.)
CREATE INDEX CONCURRENTLY On a live PostgreSQL table, build the new index without blocking writes (it takes longer and uses more resources).
BATCH AND PARALLELIZE Large batches, several concurrent writers within the target's limits, backoff on throttling.
DISABLE REPLICATION/REFRESH Where the engine allows, reduce refresh intervals and replica sync during the load, then restore.
SIZE THE TARGET FOR THE LOAD Build time is memory-bound; an undersized machine pages and the build takes days.
CHECK RESULTS A rebuilt index has different random structure; expect small recall differences — measure, don't assume.
-- PostgreSQL: build a replacement HNSW index without blocking production, then retire the old one
SET maintenance_work_mem = '8GB';
SET max_parallel_maintenance_workers = 7;
CREATE INDEX CONCURRENTLY chunks_embedding_hnsw_v2 ON chunks USING hnsw (embedding vector_cosine_ops) WITH (m = 32, ef_construction = 200);
-- ... validate recall/latency against the benchmark, then:
DROP INDEX CONCURRENTLY chunks_embedding_hnsw;
Pitfalls specific to engine-to-engine moves
METRIC / NORMALIZATION DIFFERENCES Engines name and implement metrics differently (cosine "similarity" vs "distance"; dot product vs negative inner product; some auto-normalize). Verify score semantics and thresholds.
FILTER SYNTAX AND SEMANTICS OData vs SQL vs JSON filters; null handling; case sensitivity; array fields; operator limits. Port and test every filter your application uses.
DEFAULTS DIFFER HNSW parameters, quantization, refresh intervals, consistency models — set them explicitly on the target rather than inheriting defaults.
LIMITS Maximum dimensions, payload sizes, index counts, k, filter complexity, batch sizes.
HYBRID / FUSION BEHAVIOR The keyword leg's analyzers and BM25 variants differ; RRF implementations and parameters differ; hybrid results will not match exactly.
ID AND PAYLOAD MODEL ID types and uniqueness, nested payloads, maximum field counts.
CONSISTENCY MODEL Read-your-writes, refresh delays, and eventual consistency change application behavior (tests that "upsert then immediately search" may break).
SECURITY MODEL RBAC, key scopes, network controls, and encryption options differ; re-implement and re-verify tenant isolation and ACL enforcement on the new engine.
COST MODEL Re-forecast: per-vector storage, per-query/RU charges, replicas, egress.
12. Vector Search Performance Tuning
Know your targets, then tune against them
TARGETS (define before tuning)
LATENCY p50 / p95 / p99 of the vector query itself (and of end-to-end retrieval — the embedding call is often a larger share than the search!)
RECALL recall@k at a stated level (for example >= 0.95 at k = 10), including FILTERED recall
THROUGHPUT queries per second at that latency, under realistic concurrency
FRESHNESS time from write to searchable
MEMORY/COST index footprint, instance size, replicas
BUILD/REBUILD TIME
THE PROCESS: measure baseline -> change ONE thing -> re-measure recall AND latency -> keep what moves you toward the target. Never tune latency without watching recall — you can always "speed up" by returning worse answers.
Build the measurement harness first
public record SweepPoint(int EfSearch, double Recall, double P50Ms, double P95Ms, double Qps);
public interface IAnnSearch { Task<IReadOnlyList<int>> SearchAsync(float[] query, int k, int efSearch); }
// Sweep the main runtime knob against exact ground truth to find the recall/latency frontier on YOUR data and queries
public static async Task<List<SweepPoint>> SweepAsync(
IAnnSearch ann, IReadOnlyList<float[]> queries, IReadOnlyList<IReadOnlyList<int>> exactTopK, int k, IEnumerable<int> efValues)
{
var points = new List<SweepPoint>();
foreach (int ef in efValues)
{
var latencies = new List<double>(queries.Count);
double recallSum = 0;
var total = Stopwatch.StartNew();
for (int i = 0; i < queries.Count; i++)
{
var sw = Stopwatch.StartNew();
IReadOnlyList<int> approx = await ann.SearchAsync(queries[i], k, ef);
sw.Stop();
latencies.Add(sw.Elapsed.TotalMilliseconds);
recallSum += RecallAtK(approx, exactTopK[i], k);
}
total.Stop();
latencies.Sort();
points.Add(new SweepPoint(ef, recallSum / queries.Count,
Percentile(latencies, 0.50), Percentile(latencies, 0.95), queries.Count / total.Elapsed.TotalSeconds));
}
return points;
}
static double Percentile(List<double> sorted, double p) => sorted[(int)Math.Min(sorted.Count - 1, Math.Floor(p * sorted.Count))];
// Read the result as a curve: pick the SMALLEST efSearch that meets the recall target, check its p95 against the SLO.
// Repeat under concurrency, with filters at several selectivities, with cold and warm caches, and with writes happening.
The tuning levers, in rough order of leverage
1. FIT THE INDEX IN MEMORY HNSW traversal is random access. If the index doesn't fit in RAM (or the OS/page cache), latency collapses. Check memory first: reduce dimensions, quantize, add RAM, or shard.
2. RUNTIME RECALL KNOB efSearch (HNSW) or nprobe/probes (IVF): free to change, no rebuild. Set it from the sweep, scaled to k.
3. REDUCE DIMENSIONS Matryoshka truncation or a smaller model (Section 1): cuts memory, bandwidth, and per-comparison cost proportionally. Validate quality on the benchmark.
4. QUANTIZE + RESCORE Scalar (int8), binary, or PQ compression with oversampling and full-precision rescoring (below): big memory and speed gains for a small, recoverable recall cost.
5. RETURN LESS Smaller k (and "top" vs. "candidates" settings); retrieve payloads only for the final few IDs, not for every candidate (large text payloads in the search response slow everything).
6. FILTER STRATEGY Selectivity-aware planning (Section 6): exact scan for tiny candidate sets; indexed filter fields; partitions/partial indexes for hot filters.
7. BUILD PARAMETERS Higher M / efConstruction when recall plateaus below target at high efSearch (rebuild required).
8. HARDWARE Faster memory bandwidth, more cores (SIMD distance kernels), NVMe for disk-oriented indexes; GPU-accelerated build and search at very large scale.
9. SHARDING AND REPLICAS Shard for DATA SIZE (and parallel build); replicate for QUERY THROUGHPUT and availability. Each shard returns its own top-k which are merged: more shards mean more fan-out and tail latency.
10. CACHE Cache query embeddings, retrieval results (ACL-scoped), and rerank outputs (see Advanced RAG Architecture, Section 18). The cheapest query is the one you don't run.
11. CLIENT AND NETWORK Co-locate the app and the vector store (same region/zone), reuse connections (pooling, HTTP/2), batch queries where the API allows, avoid chatty sequences of small calls.
12. THE EMBEDDING CALL Often the largest latency component of retrieval: cache query embeddings, use a nearby/faster endpoint or a smaller model, and overlap it with other work (run the keyword leg in parallel).
Quantization with rescoring: binary + full-precision rerank
// Phase 1: cheap Hamming scan over 1-bit vectors (1,536-d float32 = 6,144 B -> 192 B). Phase 2: re-score the oversampled shortlist with full-precision vectors.
public static ulong[] BinaryQuantize(float[] v)
{
var bits = new ulong[(v.Length + 63) / 64];
for (int i = 0; i < v.Length; i++)
if (v[i] > 0) bits[i >> 6] |= 1UL << (i & 63); // keep only the SIGN of each dimension
return bits;
}
public static int Hamming(ulong[] a, ulong[] b)
{
int distance = 0;
for (int i = 0; i < a.Length; i++) distance += BitOperations.PopCount(a[i] ^ b[i]); // XOR + popcount: extremely fast
return distance;
}
public static List<(int Id, float Score)> TwoPhaseSearch(
IReadOnlyList<ulong[]> binaryVectors, IReadOnlyList<float[]> fullVectors, float[] query, int k, int oversample = 10)
{
ulong[] qBits = BinaryQuantize(query);
// Phase 1: approximate shortlist of k x oversample candidates by Hamming distance
int[] shortlist = Enumerable.Range(0, binaryVectors.Count)
.OrderBy(i => Hamming(binaryVectors[i], qBits))
.Take(k * oversample)
.ToArray();
// Phase 2: exact re-scoring of ONLY the shortlist with full-precision vectors (kept in slower/cheaper storage in a real system)
return shortlist.Select(i => (Id: i, Score: TensorPrimitives.Dot(fullVectors[i], query)))
.OrderByDescending(x => x.Score).Take(k).ToList();
}
// Measure recall of this two-phase search vs. exact for oversample = 1, 3, 5, 10 on your data: binary quantization works best on high-dimensional embeddings, and the right oversampling factor is empirical.
Engine-specific tuning notes
POSTGRESQL / PGVECTOR
- Confirm the index is USED: EXPLAIN (ANALYZE, BUFFERS) shows an Index Scan on the HNSW/IVFFlat index; it needs ORDER BY <distance operator matching the opclass> LIMIT n.
- Runtime knobs: hnsw.ef_search, ivfflat.probes; filtering: hnsw.iterative_scan / hnsw.max_scan_tuples (0.8+).
- Memory: shared_buffers and the OS cache must hold the index; pg_prewarm can load it after restart; maintenance_work_mem and parallel maintenance workers speed index builds.
- Smaller footprints: halfvec (16-bit floats) and bit/binary-quantized expression indexes with a re-ranking query (pgvector 0.7+).
- Keep statistics fresh (ANALYZE); consider partitioning for tenant-heavy workloads; watch VACUUM/REINDEX after churn.
AZURE AI SEARCH
- Vector profiles bind a field to an algorithm configuration (HNSW or exhaustive KNN) and optional compression (scalar or binary quantization with oversampling and rescoring against original vectors).
- Use exhaustive KNN for small or highly selective scopes; choose the vector filter mode (pre- vs. post-filter) deliberately; tune k vs. top; scale with replicas (QPS) and partitions (size); semantic ranking
and hybrid queries add latency — measure them separately.
- Smaller dimensions and quantization directly reduce the (tier-limited) vector index quota you consume.
GENERIC ENGINES (Qdrant, Weaviate, Milvus, Elasticsearch/OpenSearch, MongoDB Atlas, Redis)
- Learn each engine's equivalents of efSearch/nprobe, quantization, payload/filter indexes, segment/merge settings, and how it combines filters with ANN. Tune with the same sweep harness — the methodology is engine-independent.
// Azure AI Search vector configuration (illustrative; property names change across API versions — check current documentation)
"vectorSearch": {
"algorithms": [
{ "name": "hnsw-cosine", "kind": "hnsw",
"hnswParameters": { "m": 8, "efConstruction": 400, "efSearch": 500, "metric": "cosine" } },
{ "name": "exact-cosine", "kind": "exhaustiveKnn", "exhaustiveKnnParameters": { "metric": "cosine" } }
],
"compressions": [
{ "name": "sq8", "kind": "scalarQuantization", "rerankWithOriginalVectors": true, "defaultOversampling": 4,
"scalarQuantizationParameters": { "quantizedDataType": "int8" } }
],
"profiles": [ { "name": "default", "algorithm": "hnsw-cosine", "compression": "sq8" } ]
}
What to monitor in production
QUERY LATENCY (p50/p95/p99) of the vector call and of end-to-end retrieval, by filter-selectivity bucket
RECALL CANARY — a fixed query set with exact ground truth, run on a schedule against production; alert on drops (index drift after churn, engine upgrades, parameter changes)
QPS, concurrency, queueing, errors and throttling (429s), timeouts
MEMORY: index size vs. available RAM, page-cache hit rate, swap activity; disk I/O for disk-based indexes
INDEX HEALTH: deleted-to-live ratio, segment/merge backlog, index build/rebuild status, freshness lag (write to searchable), ingestion lag
RESULT QUALITY SIGNALS: fewer-than-k result rate (filter starvation), empty-result rate, score distribution drift
COST: storage, compute, replicas, embedding spend, per-tenant usage
(See AI Observability for tracing and metrics conventions.)
13. Vector Search Security
The vector store holds your most sensitive data in a new form
A vector database stores embeddings derived from your documents, usually alongside the text itself and metadata like ACLs. It is reachable by services that take untrusted input, and it has a query interface that returns "the nearest things to whatever you ask." All the security fundamentals apply (see AI Security); this section covers what is specific to vector stores.
Threats
CROSS-TENANT / CROSS-USER LEAKAGE A missing or bypassable filter returns another tenant's or user's content. The #1 vector-store incident class. Includes filtered-ANN edge cases and post-filter shortcuts.
UNAUTHENTICATED OR EXPOSED ENDPOINTS Vector databases (and their dashboards) bound to public interfaces with no authentication or default credentials — a recurring real-world exposure. Anyone who can query can read.
EXCESSIVE PRIVILEGE One credential that can read, write, delete, and administer — so a bug or compromise in the query service can wipe or poison the index.
EMBEDDING INVERSION / DATA RECOVERY Research shows that text can be partially reconstructed from embeddings; vectors alone are NOT anonymized. Treat them as sensitive as the source.
MEMBERSHIP / ATTRIBUTE INFERENCE Probing nearest neighbors can reveal whether specific content (a person's record, a confidential document) is in the index.
CORPUS POISONING An attacker who can add content (a wiki page, an uploaded file, a crawled page) crafts text that embeds close to many queries ("hub" documents) or carries prompt-injection payloads, so it is retrieved
and influences answers (see AI Security, Section 2).
FILTER / QUERY INJECTION Filters built by concatenating user input into OData, SQL, or JSON filter strings allow filter bypass or errors.
RESOURCE EXHAUSTION Huge k, expensive filters, very high efSearch/ef, unbounded batch queries, or rapid-fire requests degrade or take down the index (and run up cost).
EXFILTRATION BY QUERY An attacker with query access (or via a prompt-injected agent) can systematically scrape the index by issuing many varied queries.
SUPPLY CHAIN Compromised client libraries, container images, or models; unreviewed extensions.
BACKUP / SNAPSHOT / LOG EXPOSURE Snapshots, embedding lakes, and logs/traces containing payloads and queries with weaker access control than the live system.
INSIDER / OVERBROAD ADMIN ACCESS Operators who can read all tenants' data.
Defenses
AUTHENTICATION AND NETWORK
- Never expose a vector store to the public internet. Private endpoints/VNet integration, firewall allow-lists, TLS everywhere. Change default ports/credentials; disable anonymous access and unused dashboards.
- Prefer identity-based authentication (Entra ID/managed identity) over static keys; if keys are used, store them in a vault, scope them, rotate them (see AI Security, Section 10).
LEAST PRIVILEGE
- Separate identities for QUERY (read-only, scoped to one index/namespace), INGESTION (write to specific indexes), and ADMIN (schema, deletion, key management). The query service must not be able to delete or reconfigure.
- Per-tenant scoped credentials where the engine supports them.
ACCESS CONTROL IN THE QUERY, AND AGAIN AT HYDRATION
- Derive tenant and ACL filters from the authenticated identity on the server and apply them INSIDE the vector query (pre-filter). Never post-filter for security, never take the tenant from the request or the model.
- Defense in depth: store only IDs (and minimal metadata) in the vector index, and fetch the actual text from an authorized system of record AFTER re-checking the user's permission. A vector-index bug then can't hand out content by itself.
- Enforce isolation structurally where possible (silo/partition per tenant, row-level security) so a coding mistake is not a breach (Section 8).
INPUT HARDENING
- Typed filter objects, allow-listed fields and operators; ESCAPE or parameterize any value that must appear in a filter expression; cap k, filter complexity, query size, and ef/probe settings server-side.
- Rate limit per user/tenant and detect scraping patterns (see AI Security, Section 11).
DATA PROTECTION
- Encryption in transit and at rest (customer-managed keys where policy demands); encrypt and access-control snapshots and the embedding lake the same as the source.
- Minimize what the index stores (don't copy sensitive fields into payloads "for convenience"); redact PII before embedding where the use case allows (see AI Security, Section 5).
- Retention and deletion that actually purges (Section 9).
CORPUS INTEGRITY
- Control who can write to the index; record provenance (source, author, ingestion time); scan and sanitize ingested content (hidden text, instructions); quarantine suspicious documents; allow rapid removal by source.
- Treat retrieved text as UNTRUSTED when it reaches a model (delimiting, least-privilege tools — see AI Security, Section 2, and Advanced RAG Architecture, Section 2).
- Monitor for "hub" documents that appear in an abnormally large share of results.
AUDIT AND DETECTION
- Log who queried which index with what filters and how many results (metadata, not content); alert on unusual query volume, new filters, bulk reads, access from unexpected locations, and admin operations.
- Test tenant isolation continuously (Section 8) and run adversarial queries as part of security testing (see AI Security, Section 14).
SUPPLY CHAIN
- Pin and scan client libraries and images; review extensions; verify model and artifact checksums (see AI Security, Section 12).
A hardened query wrapper
public sealed class SecureVectorSearch(IVectorIndex index, ILogger<SecureVectorSearch> log)
{
private const int MaxK = 50;
private static readonly HashSet<string> AllowedDocTypes = ["policy", "manual", "faq", "runbook"];
public async Task<IReadOnlyList<Hit>> SearchAsync(UserContext user, float[] queryVector, SearchRequest request, CancellationToken ct)
{
int k = Math.Clamp(request.Top, 1, MaxK); // cap result size server-side
// SECURITY filters come from the authenticated identity ONLY. Group IDs should be GUIDs/opaque IDs without commas.
var clauses = new List<string>
{
$"tenantId eq {ODataQuote(user.TenantId.ToString())}",
$"group_ids/any(g: search.in(g, {ODataQuote(string.Join(",", user.GroupIds))}, ','))" // the Azure AI Search security-trimming pattern
};
// BUSINESS filters are allow-listed and escaped — never concatenated from raw input
if (request.DocType is not null)
{
if (!AllowedDocTypes.Contains(request.DocType)) throw new ArgumentException("Unsupported document type.");
clauses.Add($"docType eq {ODataQuote(request.DocType)}");
}
string filter = string.Join(" and ", clauses);
IReadOnlyList<Hit> hits = await index.SearchAsync(queryVector, k, filter, ct); // the filter is applied INSIDE the query (pre-filter)
// Defense in depth: hydrate text from the system of record, re-checking the user's permission on each item
var authorized = new List<Hit>();
foreach (Hit hit in hits)
if (await documents.UserCanReadAsync(user, hit.DocId, ct)) authorized.Add(hit);
log.LogInformation("Vector search by {User} tenant {Tenant}: {Returned}/{Requested} authorized", user.Id, user.TenantId, authorized.Count, k); // metadata only, never query text
return authorized;
}
// OData string literals escape a single quote by doubling it
private static string ODataQuote(string value) => "'" + value.Replace("'", "''") + "'";
}
Treat embeddings as data subject to the same obligations
CLASSIFICATION Vectors inherit the classification of their source (confidential text -> confidential vectors).
RESIDENCY Where vectors are stored and processed is subject to the same residency rules.
SUBJECT RIGHTS Access, correction, and erasure requests apply to embeddings and derived payloads (Section 9).
THIRD PARTIES Hosted embedding APIs and hosted vector databases are data processors: review retention, training use, sub-processors, and regions.
LOGGING Don't log raw vectors or queries containing sensitive text; redact before capture (see AI Observability, Section 12).
14. Capacity Planning
Do the arithmetic before you buy hardware
public static class VectorCapacity
{
private const double GiB = 1024.0 * 1024 * 1024;
// Raw vector storage: N x dimensions x bytes per dimension
public static double RawVectorsGiB(long n, int dims, double bytesPerDim) => n * dims * bytesPerDim / GiB;
// HNSW links: layer 0 holds up to 2*M neighbor IDs per node (4 bytes each), plus a few percent for upper layers and bookkeeping.
// Real engines add their own overhead (metadata, deleted flags, ID maps); treat this as a floor.
public static double HnswGraphGiB(long n, int m) => n * (2.0 * m * 4) * 1.10 / GiB;
// IVF-Flat: full vectors + one list entry per vector + centroids
public static double IvfFlatGiB(long n, int dims, double bytesPerDim, int nlist) =>
(n * dims * bytesPerDim + n * 8.0 + (double)nlist * dims * 4) / GiB;
// Product quantization: m sub-vector codes of `bitsPerCode` bits per vector (excludes the full-precision vectors kept for rescoring)
public static double PqGiB(long n, int subvectors, int bitsPerCode = 8) => n * subvectors * bitsPerCode / 8.0 / GiB;
}
EXAMPLES (illustrative)
10 M vectors x 1,536-d float32: raw vectors 57.2 GiB; HNSW (M = 16) links ≈ 1.3 GiB -> the graph is only ~2% on top of the vectors at this dimensionality
10 M vectors x 128-d float32: raw vectors 4.8 GiB; HNSW (M = 16) links ≈ 1.3 GiB -> ~27%: at LOW dimensionality the graph overhead is significant
100 M vectors x 1,536-d: float32 572 GiB int8 143 GiB binary / PQ(192 B) 17.9 GiB (+ full-precision vectors on cheaper storage if rescoring)
REAL-WORLD HEADROOM (multiply the floor by...)
x 1.5-2.0 for engine overhead, metadata/payload indexes, fragmentation, segment merges, and growth between rebuilds
x replicas each replica holds a full copy for availability and read throughput
+ working memory for queries, filters, and (importantly) index builds/rebuilds, which can need substantially more RAM than serving
+ OS page cache so the index stays memory-resident
PLAN CAPACITY FOR NEXT YEAR'S DATA, not today's — rebuilding a full index under pressure is the expensive way to learn you under-provisioned.
Levers that change the arithmetic
FEWER DIMENSIONS 1,536 -> 768 halves vector memory and per-comparison cost.
SMALLER NUMERIC TYPE float32 -> float16/int8/binary: 2x / 4x / 32x, with rescoring to recover recall.
TIERING Hot data in RAM at full precision, warm quantized or on disk, cold archived (Section 9).
DEDUPLICATION Near-duplicate chunks and documents waste vectors (and result slots): deduplicate at ingestion.
CHUNKING CHOICES Larger chunks (or parent-child indexing of small children only) change vector counts by multiples.
SHARDING Splits memory across nodes (adds fan-out); does not reduce total memory.
15. Choosing and Operating a Vector Database
Engineering questions that decide the choice
SCALE AND MEMORY How many vectors, what dimensions, what growth? Does the engine support quantization and disk-oriented indexes if RAM cost bites?
INDEX TYPES HNSW only, or also IVF/PQ, flat, and disk-based graphs? Can you tune parameters per index/query?
FILTERING How does it combine filters with ANN — pre-, post-, or in-graph? Indexed payload fields? Behavior at selective filters (test it)?
HYBRID Native keyword + vector with fusion? Sparse vectors? Multi-vector fields?
MULTI-TENANCY What primitives exist (namespaces, partitions, per-tenant shards, per-tenant indexes) and what are the limits?
UPDATES AND DELETES Real-time upserts? How are deletes handled and compacted? Freshness (write-to-searchable lag)?
CONSISTENCY AND DURABILITY Replication, snapshots, restore tooling, RPO/RTO.
SECURITY Identity-based auth, RBAC granularity, private networking, encryption/CMK, audit logs.
OPERATIONS Managed vs. self-hosted, upgrade story, monitoring/metrics, cost model (per vector, per query/RU, per replica).
ECOSYSTEM .NET SDK quality, integration with Microsoft.Extensions.VectorData / Semantic Kernel connectors, your existing database or cloud.
A rough feature map (verify against current documentation — these evolve)
POSTGRESQL + PGVECTOR HNSW and IVFFlat; halfvec and binary-quantization options; iterative filtered scans; SQL joins, transactions, RLS, partitioning. Strong when vectors belong next to relational data; RAM-bound at very large scale.
AZURE AI SEARCH HNSW and exhaustive KNN; scalar/binary quantization with rescoring; native hybrid (BM25 + vector, RRF) and semantic ranking; filterable fields and vector filter modes; replicas/partitions scale; managed.
SQL SERVER / AZURE SQL Native vector data type and distance functions, with approximate (DiskANN-style) vector indexes arriving in recent versions — useful when vectors should live in the relational engine you already run.
AZURE COSMOS DB Flat, quantized-flat, and DiskANN vector indexes inside a globally distributed document database.
MONGODB ATLAS Vector search index over document fields, with filter fields, quantization options, and Atlas Search for hybrid.
REDIS In-memory HNSW and FLAT with tag/numeric/text filtering; lowest latency, memory-priced.
ELASTICSEARCH / OPENSEARCH HNSW alongside mature inverted-index search, filtering, and aggregations; quantization options.
DEDICATED VECTOR DATABASES Qdrant (payload indexes, named and sparse vectors, quantization), Weaviate (multi-tenancy objects, hybrid), Milvus (many index types incl. disk and GPU, partition keys), Pinecone (managed, namespaces).
Choose the one closest to your existing data and operational skills that passes YOUR benchmark on filtered recall, latency, and cost — then abstract it behind your own interface so the decision stays reversible.
An operating checklist
DESIGN Embedding spec recorded; dimensions chosen and justified; metric/normalization documented; tenancy model decided; filter fields indexed; index parameters set explicitly (no silent defaults).
MEASURE Benchmark queries + exact ground truth; recall@k including filtered; latency percentiles under load; recall canary scheduled in production.
SECURE Private networking; identity auth; separate read/write/admin identities; tenant/ACL filters inside queries; ID-only index with authorized hydration; input caps and rate limits; encryption; audit logging.
OPERATE Index health metrics (deleted ratio, merges, freshness); compaction/reindex schedule; backups AND tested restores; embedding lake for rebuilds; documented rebuild runbook with timing.
CHANGE Blue/green indexes and alias swaps for parameter, model, and engine changes; shadow reads; staged cutover; rollback window.
GOVERN Retention and erasure procedures covering vectors, payloads, caches, and backups; tenant offboarding verified; cost attribution; capacity forecast for the next 12 months.
16. Interview-Style Questions
Concise model answers — expand with examples from your own experience.
1. Cosine similarity, dot product, and Euclidean distance — when do they give the same ranking?
When vectors are normalized to unit length: ‖a−b‖² = 2 − 2·cos(a,b) and dot = cosine, so all three rank neighbors identically. For unnormalized vectors they differ — cosine ignores magnitude, dot product and L2 don't — so use the metric the model was trained for, and prefer dot product on normalized vectors because it's cheapest.
2. What does an HNSW index do, and what do M, efConstruction, and efSearch control?
HNSW is a layered proximity graph: sparse upper layers give long hops, layer 0 holds all vectors; search descends greedily then runs a beam search of width efSearch. M is the number of links per node (recall vs. memory/build time), efConstruction is the build-time candidate width (graph quality vs. build cost), efSearch is the query-time width (recall vs. latency) and is the knob to tune first.
3. HNSW versus IVF — how do you choose?
HNSW generally gives the best recall/latency when the index fits in RAM, builds incrementally, but is memory-heavy, slow to build, and awkward for deletes and filters. IVF (especially with PQ/SQ) is smaller, faster to build, handles deletes and filters naturally, and scales beyond RAM, but needs representative training, drifts, and usually trails HNSW at equal recall. Start with HNSW at RAG scale; consider IVF-PQ or disk-based graphs when memory cost dominates.
4. Why does adding a filter hurt approximate-search recall, and what do you do about it?
The index is built over all vectors; a selective filter removes most candidates from a post-filtered ANN result (too few results) or disconnects the graph traversal. Use pre-filtering with an exact scan when the candidate set is small, filter-during-search with a wider ef when moderate, indexed filter fields, partitions/partial indexes for hot filters like tenant, iterative scans (pgvector 0.8+), and always measure filtered recall at several selectivities.
5. How would you design multi-tenancy for a vector store?
Choose by tenant count and isolation needs: pooled (tenant filter) for many small tenants, partitions/namespaces as a middle path, index per tenant for stronger isolation and per-tenant keys/models, deployment per tenant for regulated customers — often tiered. Enforce isolation structurally (tenant from identity in a repository layer, row-level security or per-tenant credentials), route small tenants to exact search, add noisy-neighbor limits, and test cross-tenant leakage in CI.
6. How do you handle deletes and updates in an ANN index?
Updates are delete + insert; deletes are tombstoned and excluded from results but leave the graph/lists dirty until compaction, vacuum, or reindex, so heavy churn wastes memory and degrades recall. Monitor the deleted ratio, schedule maintenance or rebuild-and-swap, make upserts idempotent and version-checked, and remember soft deletes don't satisfy erasure obligations.
7. How do you migrate to a new embedding model with no downtime?
Treat it as a new vector space: build a new index (or new named vector field) with the new spec, backfill from source or the embedding lake within provider rate limits, evaluate on the benchmark, shadow traffic, switch an alias, keep the old index for rollback, and recalibrate similarity thresholds. Never mix specs in one ANN search; stamp every vector and trace with the embedding spec.
8. How do you migrate between vector engines?
Copy existing vectors and payloads rather than re-embedding; dual-write via change feed, backfill with resumable batched upserts, bulk-load first and index after, verify counts plus search quality (recall vs. exact, overlap@k with the old system), shadow reads, staged cutover, rollback window. Re-verify metric semantics, filter syntax, defaults, hybrid behavior, consistency model, and the security/tenant-isolation implementation on the new engine.
9. A vector query is slow. How do you diagnose and tune it?
Measure first: is time in the embedding call, the search, payload retrieval, or the network? Check that the index fits in memory and is actually used (EXPLAIN), then sweep efSearch/nprobe against exact ground truth to pick the smallest setting meeting the recall target, and consider fewer dimensions, quantization with rescoring, smaller k, fetching payloads only for the final IDs, filter-aware planning, caching, co-location, and replicas for throughput. Always check recall alongside latency.
10. What is quantization with rescoring and why use it?
Compress vectors (int8, binary, PQ) to cut memory 4–32× and speed up approximate scoring, retrieve an oversampled shortlist (3–10× k), then re-score only those with full-precision vectors to recover recall. It makes large indexes affordable; the oversampling factor and quality loss must be measured on your data.
11. What are the security risks specific to a vector database?
Cross-tenant leakage through missing or bypassable filters, exposed unauthenticated endpoints, overbroad credentials, embedding inversion (vectors aren't anonymous), corpus poisoning and prompt injection via retrieved chunks, filter injection, resource exhaustion and scraping via queries, and unprotected snapshots/logs. Mitigate with private networking and identity auth, least-privilege split identities, ACL filters inside the query plus authorized hydration from the system of record, typed/escaped/allow-listed filters, caps and rate limits, encryption, provenance controls, and audit logging.
12. How big will my index be?
Vectors: N × dimensions × bytes per dimension (1M × 1,536 × float32 ≈ 5.7 GiB); HNSW adds roughly 2M×4 bytes per vector for links (small at high dimensions, significant at low); add 1.5–2× for engine overhead and growth, multiply by replicas, and reserve extra working memory for rebuilds. Reduce with fewer dimensions, quantization, and tiering.
13. Why can't you compare vectors from two different embedding models?
Each model defines its own geometry; coordinates and distances have no shared meaning, so cross-model similarity is noise that still returns plausible-looking numbers. That is why the embedding spec (model, version, dimensions, normalization, preprocessing) is stored with every vector and checked at query time.
17. Common Pitfalls
| Pitfall | Why it hurts | Better approach |
|---|---|---|
| Querying with a different metric than the index was built for | Index unused (full scan) or wrong neighbors | Match operator/metric to the index's operator class; normalize on write (Section 2) |
| Using dot product on unnormalized text embeddings | Vector magnitude, not meaning, drives rankings | Normalize, or use cosine (Section 2) |
| Mixing normalized and unnormalized vectors | Dot/L2 rankings break silently | Enforce and test normalization in ingestion (Section 2) |
| Assuming more dimensions means better retrieval | Higher cost with little or no gain | Benchmark smaller dimensions / Matryoshka truncation (Section 1) |
| Truncating a model not trained for it | Large quality loss | Truncate only models that support it; re-normalize (Section 1) |
| Trusting default HNSW/IVF parameters | Wrong recall/latency for your data | Sweep runtime knobs; rebuild with tuned M/efConstruction if recall plateaus (Sections 4, 12) |
| Judging an index by unfiltered recall | Filtered recall can be far worse | Measure recall at several filter selectivities (Section 6) |
| Post-filtering for access control | Too few results and potential leakage | Pre-filter inside the query; structural isolation (Sections 6, 13) |
| Unindexed filter fields | Metadata scans dominate latency | Declare and index filterable fields (Section 6) |
| Letting the HNSW index spill out of RAM | Latency cliff | Size memory with headroom; quantize or shard (Sections 4, 14) |
| Ignoring tombstones and churn | Bloat, slower queries, degraded recall | Monitor deleted ratio; schedule compaction/reindex; alias-swap rebuilds (Section 9) |
| Treating soft delete as erasure | Data still stored; compliance failure | Hard delete across index, payloads, caches, backups (Section 9) |
| Dual-writing from application code | Index and source diverge on partial failure | Outbox/CDC with idempotent, version-checked upserts plus reconciliation (Section 9) |
| No embedding spec stored with vectors | Can't tell what space vectors live in; unsafe migrations | Store spec ID per vector and in index metadata; check at query time (Section 10) |
| Mixing embedding versions in one ANN search | Meaningless similarity scores | Separate indexes/fields per spec; blue/green cutover (Section 10) |
| Re-embedding for an engine migration | Slow, costly, subtly different vectors | Copy existing vectors; use an embedding lake (Sections 9, 11) |
| Building indexes incrementally during bulk loads | Days-long builds | Load first, build index after, with tuned build memory/parallelism (Section 11) |
| Cutting over without comparing search behavior | Silent quality regressions | Shadow reads, overlap@k, recall vs. exact, filter parity (Section 11) |
| Tuning latency without watching recall | "Faster" because worse | Always sweep recall and latency together against exact ground truth (Section 12) |
| Pulling large payloads for every candidate | Slow queries and heavy bandwidth | Return IDs from search; hydrate only the final few (Sections 12, 13) |
| One multi-tenancy model for all tenants | Small-tenant recall problems or runaway per-tenant overhead | Tiered model; exact search or partitions for small tenants; silo for large/regulated ones (Section 8) |
| Session-level tenant settings on pooled connections | Tenant context leaks into the next request | Transaction-local settings (set_config(..., true)) with row-level security (Section 8) |
| Exposed vector DB ports and dashboards | Direct data exposure | Private networking, authentication, no anonymous access (Section 13) |
| Concatenating user input into filter strings | Filter bypass and injection | Typed filters, allow-lists, escaping (Section 13) |
| Treating embeddings as anonymous | Inversion and inference risks; compliance gaps | Classify vectors like their source; same controls, residency, and erasure (Section 13) |
| No recall canary in production | Silent drift after churn, upgrades, or tuning | Scheduled exact-ground-truth canary with alerts (Section 12) |
| Capacity planned for today's data | Emergency rebuilds under load | Forecast 12 months, include headroom, replicas, and rebuild memory (Section 14) |
Quick Reference Table
| Topic | Key idea | Practical takeaway |
|---|---|---|
| Dimensions | Fixed by the embedding model; cost = d × bytes × N | 1,536-d float32 ≈ 6 KiB/vector; test smaller dims (Matryoshka) |
| Metrics | Cosine = dot = L2 ranking on unit vectors | Normalize on write; use the model's metric; match index and query metrics |
| ANN | Trade recall for speed and memory | Measure recall@k vs. exact on your data; use exact search for small N and tiny filtered sets |
| HNSW | Layered proximity graph | Tune efSearch first; M and efConstruction at build; RAM-resident; deletes and filters are hard |
| IVF | k-means cells + nprobe scan | Smaller and faster to build; needs training; pairs with PQ/SQ and rescoring |
| Quantization | int8 / binary / PQ compression | 4–32× smaller; oversample and rescore with full precision |
| Filtering | Pre-, post-, or in-graph | Selectivity decides; exact scan for tiny sets; verify filtered recall; index filter fields |
| Hybrid indexes | Inverted index + ANN (+ sparse/multi-vector) | One engine simplifies filters and consistency; fuse by rank (RRF) |
| Multi-tenancy | Pooled, partitioned, silo, deployment-per-tenant | Tier by tenant size/isolation needs; enforce structurally; test leakage |
| Data lifecycle | Derived, rebuildable data | Idempotent version-checked sync; tombstones; compaction; retention and real erasure; embedding lake |
| Embedding versioning | Spec = model + version + dims + normalization + preprocessing | Store spec per vector; blue/green or named vectors; never mix specs |
| Migrations | Move vectors, don't recompute | Dual-write, resumable backfill, load-then-index, shadow reads, staged cutover |
| Performance tuning | Measure → change one thing → re-measure recall and latency | Memory fit, efSearch/nprobe, dims, quantization, smaller k, caching, replicas |
| Security | Isolation, exposure, inversion, poisoning, injection | Private + identity auth, split identities, ACL in query + authorized hydration, typed filters, audit |
| Capacity | N × d × bytes (+ graph + overhead × replicas) | Forecast growth; headroom 1.5–2×; rebuild memory |
Conclusion
A vector database looks simple from the outside — put vectors in, get neighbors out — but each layer has engineering consequences. Dimensions set memory, bandwidth, and per-query cost, and are one of the cheapest things to shrink. The three similarity metrics collapse into one on normalized vectors, so the practical rules are to normalize on write, use the model's metric, and never query with a metric the index wasn't built for. Approximate search buys speed by giving up some recall: HNSW does it with a layered graph you tune mainly through efSearch (and M and efConstruction at build time), IVF with k-means cells and nprobe, and quantization with compression plus full-precision rescoring. All of them are statements about your data, which is why recall must be measured against exact ground truth — including filtered recall, because filters, tenancy, deletes, and churn are exactly where approximate indexes quietly degrade.
The remaining discipline is operational. Choose a multi-tenancy model deliberately and enforce isolation structurally so a coding slip isn't a breach. Treat the index as derived data with a lifecycle: idempotent, version-checked synchronization from the source, tombstones and compaction, retention and genuine erasure, and a durable embedding lake so a rebuild doesn't mean re-buying every embedding. Treat the embedding spec as part of every vector's identity, and change it with blue/green indexes or named vectors, never by mixing spaces. Run migrations by copying vectors, dual-writing, backfilling resumably, shadow-reading, and cutting over gradually with a rollback window. Tune by building the measurement harness first and changing one lever at a time — memory fit, runtime knobs, dimensions, quantization, payload size, caching, replicas — always reading recall and latency together. And secure the store like the sensitive-data system it is: private networking and identity authentication, split read/write/admin credentials, access filters inside the query plus authorized hydration, typed and escaped filters, request caps, provenance controls against poisoning, encryption, and audit logs. Do the capacity arithmetic early, keep the engine behind an interface you control, and a vector database stops being a black box that is fast in demos and fragile in production, and becomes a component you can size, tune, secure, evolve, and trust.
Found this useful? Feel free to star the repo, open an issue with corrections, or share the "recall looked perfect until we turned on the tenant filter" story that made the case for measuring filtered recall click better than any benchmark chart.
Top comments (0)