The last turn of the dial. Move the full-precision vectors to disk, stack a few more savings, and pay for the latency you actually need.
The first two articles in this series shrank the vectors and kept them in memory. 1.3 TB of RAM for 100M Vectors? Shrink Them Before They Hit OpenSearch covered pre-quantized vectors, where you convert to FP16, byte, or binary before indexing. Native Quantization: Let OpenSearch Service Compress Your Vectors handed that conversion to the engine at index time, from 16-bit scalar quantization down to 32x with optimized scalar quantization. Both kept the working set in RAM, where search answers in milliseconds.
This article takes the last turn of the dial: it moves the full-precision vectors to disk. Amazon OpenSearch Service keeps a compressed copy in memory and searches it first, then reads the full-precision vectors from disk to rescore the best candidates. For a replicated index of 100 million embeddings at 1,536 dimensions, that drops resident memory from about 1.3 TB, the best-latency deployment with everything in RAM, to roughly 66 GB. The price is some added search latency. A few stacking techniques, covered later, trim the footprint further.
On-disk mode: compressed in memory, full precision on disk
Disk-based vector search, introduced in OpenSearch 2.17, runs a search in two phases. The compressed vectors stay resident in memory, and OpenSearch searches them first to gather more candidates than you asked for. Then it loads the full-precision vectors for those candidates from disk and rescores them, returning the top k by true similarity. The large full-precision copy lives on the secondary storage tier, not in RAM, which is where the memory savings come from. Rescoring is on by default, with an oversample factor of 2.0 at the default 32x compression, so recall stays strong despite the compressed first pass.
You enable it with a single field-level parameter, mode set to on_disk. By default that selects the Faiss engine, the HNSW method, and 32x compression, which routes through the optimized scalar quantizer OpenSearch uses at that level, bringing a replicated 100-million-vector index down to roughly 66 GB resident. The mode works with the float and half_float data types. To trade some of the savings back for recall, lower the compression with compression_level; at 4x, OpenSearch uses the Lucene engine.
PUT /my-ondisk-index
{
"settings": { "index.knn": true },
"mappings": {
"properties": {
"embedding": {
"type": "knn_vector",
"dimension": 1536,
"space_type": "l2",
"data_type": "float",
"mode": "on_disk"
}
}
}
}
Latency is the thing to measure. In-memory search runs every comparison at RAM speed. On-disk search adds a second pass that reads full-precision vectors from the storage tier, so SSD throughput sets the pace. In the OpenSearch project's disk-based search benchmark on 113 million 1,024-dimension vectors, on-disk search at 32x answered in about 100 milliseconds at p90 with a single client, compared with about 24 milliseconds for the uncompressed in-memory index. Latency climbs as concurrency rises, so test at your real query load. For RAG, agent, and batch workloads that is usually a good trade. For the tightest user-facing latency budgets, keep the working set in RAM with one of the in-memory options from the earlier articles.
A useful pattern runs two indexes on one cluster: an in-memory quantized index for interactive, user-facing queries, and an on-disk index for batch or background work such as overnight RAG precomputation or periodic analytics over the full corpus. Each is sized for its own latency target, so you are not paying RAM prices for queries that can wait.
Derived source: cut disk storage, not just RAM
Derived source saves disk, not memory. Everything else in this series shrinks what sits in RAM; this setting shrinks what sits on disk. Without it, OpenSearch keeps a full copy of every vector in the stored _source field, alongside the vector data the search structures already hold, so each vector is stored twice. Derived source drops that copy from _source and rebuilds the values on demand, so the update, update_by_query, and reindex APIs keep working. Less data on disk also makes search faster: in the OpenSearch benchmarks for derived vector source, storage dropped by 3x, force merges ran about 10% faster, and search latency on the Lucene engine fell by 90%, because less unneeded vector data moves through the page cache. Derived source is on by default for vector indexes on OpenSearch 3.0 and later, and it matters most for on-disk indexes, where the full-precision vectors already occupy the storage tier. For an index created before the default, check index.knn.derived_source.enabled; the setting is static, so turning it on means reindexing.
Serving indexes larger than memory, handled for you
You do not have to load an entire Faiss index into memory to search it. Memory-optimized search, introduced in OpenSearch 3.1, memory-maps the index file and lets the operating system file cache serve reads, so a node can hold indexes larger than its physical memory and repeated queries come from cache. When you choose on-disk mode, or 32x compression with the optimized scalar quantizer, OpenSearch applies it for you behind the scenes. You can also turn it on explicitly for any Faiss HNSW index with a single index setting. On instances with large local memory and NVMe storage, memory-mapping raises effective capacity per node, with occasional p99 variability when a read misses the cache. The setting is static, so for an existing index you close the index, update the setting, and reopen it. It applies to Faiss HNSW only; IVF and product quantization indexes still load into memory.
PUT /my-vector-index
{
"settings": {
"index.knn": true,
"index.knn.memory_optimized_search": true
},
"mappings": {
"properties": {
"embedding": {
"type": "knn_vector",
"dimension": 1536,
"space_type": "l2",
"method": { "name": "hnsw", "engine": "faiss" }
}
}
}
}
IVF with product quantization for billion-scale
HNSW is the default graph for most workloads, but at billion scale even a quantized graph can exceed the memory budget. Faiss offers an alternative: an inverted file (IVF) index combined with product quantization. IVF partitions the vector space into clusters with k-means and searches only the clusters near the query, while product quantization encodes each vector into a compact set of codes. There is no navigation graph to hold, so the footprint is smaller than HNSW, at the cost of a training step to learn the cluster centroids and codebooks, and typically lower recall than HNSW at the same latency. Reach for IVF with product quantization when you have billions of vectors and cost is the binding constraint. IVF is available only on the Faiss engine; the Lucene engine supports HNSW and flat indexes but not IVF. Like all product quantization, it requires training before you index.
Reminders from earlier in the series
A few techniques from the earlier articles stack with everything here. When a filter narrows the candidates to roughly 100,000 vectors or fewer, exact k-NN skips the graph entirely and returns perfect recall with no graph to hold in memory. On nodes dedicated to vector search, raising the k-NN memory circuit breaker, knn.memory.circuit_breaker.limit, above its 50% default fits more vectors per node before you add hardware. And GPU-accelerated graph construction cuts build times for full-corpus rebuilds and embedding-model upgrades; see the GPU acceleration write-up.
The full spectrum, end to end
Here is the whole dial across the three articles, measured against the same baseline: 100 million vectors at 1,536 dimensions, HNSW with m of 16, one replica. RAM figures are the resident footprint with the replica. Dollar figures are order-of-magnitude on demand before reserved-instance or savings-plan discounts, meant to show relative magnitude, not quote a price.
| Method | Resident RAM (with replica) | p50 latency | Recall |
|---|---|---|---|
| FP32 baseline | ~1.3 TB | single-digit ms | reference |
| FP16 / SQ 16-bit | ~656 GB | single-digit ms | indistinguishable from FP32 |
| INT8 / byte | ~341 GB | single-digit ms | 0.99 on Euclidean sets |
| SQ 4-bit (8x) | ~184 GB | single-digit ms | 0.94-0.98 with rescoring |
| SQ 2-bit (16x) | ~105 GB | single-digit ms | moderate, benchmark it |
| OSQ 1-bit (32x) | ~66 GB | single-digit ms | 0.94-0.98 with rescoring |
| Product quantization | ~42 GB | single-digit ms | moderate, codec-dependent |
| On-disk (32x default) | ~66 GB resident + FP32 on disk | ~100 ms p90 (benchmark) | strong, two-phase rescoring |
Dimensionality reduction stacks on every row. If your embedding model emits shorter vectors, or you apply principal component analysis or Matryoshka truncation, cutting 1,536 dimensions to 768 roughly halves each RAM figure again.
Wrapping up the series: pick your point on the dial
Across all three articles, the choice comes down to a tradeoff among cost, latency, and accuracy. You can hold any two steady by giving up some of the third. Every step down the table saves memory and pays for it in recall or in latency, and oversampling with rescoring buys recall back at a latency cost. No setting is right for every workload; the right one is the point where your recall and latency requirements still hold at the lowest cost.
Two settings make good middle grounds. FP16, either pre-quantized half_float vectors or 16-bit scalar quantization in the engine, halves memory to about 656 GB with recall effectively indistinguishable from full precision and no change in latency. It is the lowest-risk first step. At the other end, 32x compression on disk takes the same index to about 66 GB resident, around a twentieth of the baseline, and two-phase rescoring against the full-precision vectors keeps recall strong, around 95% recall@100 in the OpenSearch disk-based search benchmark. The cost is latency, about 100 milliseconds at p90 in that benchmark, which suits RAG, agent, and batch workloads. Between the two, you can cover a wide range of workloads without exotic configuration.
You do not have to walk the dial by hand. Auto-optimize for Amazon OpenSearch Service takes your vectors from Amazon S3, along with the recall and latency you need, and evaluates HNSW parameters, quantization, and rescoring settings against them. It returns up to three recommended configurations, typically within 30 to 60 minutes. The jobs run on separate infrastructure that OpenSearch Service manages, so they don't consume resources on your domain, and each job is billed at a flat rate. That makes it a simple, cost-effective way to explore the dial on your own data before you commit to a configuration.
Whichever setting you choose, measure it. Start from a full-precision baseline on your own evaluation queries, record recall and p50 and p95 latency, and compare every option against it. The right point moves as the corpus grows, the embedding model changes, and the accuracy requirement shifts, so build recall measurement into the deployment pipeline and re-check when the inputs change.
Top comments (0)