You loaded 125 million product embeddings into your vector search cluster. The math looked fine on paper: 1536 dimensions, HNSW indexing, three availability zones. Then the first bill arrived. Eighteen thousand dollars a month. Your entire text search infrastructure costs six thousand. The vector layer alone is triple that.
The product manager asked a reasonable question: why does a column of numbers cost more than the rest of the application combined? The answer is precision nobody needs.
Think of vector storage like shipping a library across the country. You could photograph every page at 600 DPI, lossless TIFF, archival quality. Or you could photograph them at 150 DPI JPEG, more than enough for anyone who wants to read the words. The first option fills a shipping container. The second fits in a suitcase. Vector search, by default, ships the container.
Why Vectors Blow Up Your Budget
A single 1536-dimension embedding at full precision: 12,288 bytes. Each dimension is a 4-byte float. Multiply by 125 million: 1.5 terabytes of vector storage before you think about the search graph.
HNSW builds multi-layer graph structures that need RAM for fast queries. The formula: (1.1 × 4 × dimensions) + (8 × edges × vector_count). With defaults and 125 million vectors: 862 GB of RAM.
Traditional text search is disk-bound. Vector search is memory-bound. The graph lives in RAM or queries take seconds. You cannot solve this by adding nodes. More nodes = more shards = the graph splits into smaller neighborhoods = degraded recall. The scaling pattern that works for logs does not work for vectors.
The Precision You're Paying For
Embedding models default to 1536 dimensions at 32-bit float. You can choose lower-dimensional outputs and reduced precision, but mostly you take the default because models benchmark best at full width.
The question nobody asks at architecture time: how much of that precision matters for retrieval?
Retrieval is not final ranking. You are narrowing 125 million candidates to a few hundred, then rescoring with a cross-encoder or business logic. The initial pass needs the right neighborhood, not perfect ordering within that neighborhood. Binary precision (1 bit per dimension) maintains 90%+ recall for that job.
The Cost-Latency-Accuracy Dial
Amazon OpenSearch Service gives you several points on this dial:
Full precision in RAM (FP32): 862 GB RAM, single-digit ms latency, $18,000/month.
FAISS FP16 in RAM: ~430 GB, single-digit ms, negligible accuracy loss. ~$10,000/month.
INT8 or binary vectors in RAM: quarter or 32x reduction. Latency stays fast. Accuracy depends on your data — for retrieval into a reranker, most workloads absorb it.
On_disk + binary quantization: graph on SSD, 44 GB RAM, ~200 ms p50, 95% accuracy. $7,000/month. Best for RAG pipelines, agent context assembly, batch reranking.
Dimensionality: 768-dim instead of 1536 halves the footprint at every level. Stacks with all of the above.
Operational Tuning
Shard sizing: vector workloads want 50-75 GB per shard (not the 10-30 GB text search guidance). HNSW graphs work better with denser neighborhoods. Fewer, bigger shards = better recall.
Off-heap memory: OpenSearch caps heap at 32 GB. Remaining memory splits between filesystem cache and k-NN vector cache. The circuit breaker knn.memory.circuit_breaker.limit (default 50%) controls how much off-heap goes to vectors. Raise it for vector-heavy workloads:
# Give 75% of off-heap to the k-NN graph cache
knn.memory.circuit_breaker.limit: 75%
On a 128 GB node: 32 GB heap + 96 GB off-heap. At 75%, vectors get 72 GB — plenty for a quantized graph with headroom.
Run diff On the Deployment
| Configuration | RAM | Latency | Monthly Cost |
|---|---|---|---|
| Full precision (FP32) in RAM | 862 GB | <10 ms | $18,000 |
| FAISS FP16 in RAM | ~430 GB | <10 ms | ~$10,000 |
| On_disk + binary quantization | 44 GB | ~200 ms | $7,000 |
| 768-dim + FP16 in RAM | ~215 GB | <10 ms | ~$6,000 |
$11,000/month between the extremes. $132,000/year. But you do not have to choose an extreme. Pick the point on the dial that matches your latency budget.
The Actual Lesson
If your CloudWatch memory utilization sits above 60% with full-precision vectors, you have room to move down the dial. Test FP16 on a shadow index. Measure the accuracy delta on your real queries.
Back to the product manager's question: why does a column of numbers cost more than the rest of the application? Because each number is stored at 32x the precision retrieval needs, and the graph lives entirely in RAM. Fix the precision, and the column of numbers costs less than the text catalog it serves.
Top comments (0)