Canonical version: https://thelooplet.com/posts/how-to-fix-vector-search-costs-ondisk-vs-inmemory-ann-indexes

How to Fix Vector Search Costs: OnDisk vs InMemory ANN Indexes

TL;DR: When RAM pricing spikes, switch to DiskANN for large vectors and keep HNSW in‑memory for latency‑critical queries; blend both to balance cost and performance.

1. Introduction

Vector search is the backbone of modern AI‑driven products: recommendation engines, semantic text retrieval, image similarity, and even fraud detection rely on nearest‑neighbor look‑ups over millions or billions of high‑dimensional embeddings. Approximate Nearest Neighbor (ANN) indexing trades a small amount of recall for orders‑of‑magnitude speed‑ups compared with a brute‑force linear scan.

The trade‑off most teams wrestle with is cost vs. latency. Cloud providers have raised memory prices faster than compute for the past three years (2023‑2025). A single HNSW index for a 200 M‑vector, 128‑dimensional dataset can consume ~250 GB of RAM, translating into a monthly bill of several thousand dollars for a single node, not counting redundancy for high availability.

DiskANN shrinks the RAM footprint by an order of magnitude by streaming most of the graph structure from NVMe SSDs, at a modest increase in query latency (typically 10‑20 ms).

This article shows why a static “all‑in‑memory” or “all‑on‑disk” strategy no longer makes sense for production workloads. Instead, we walk through a hybrid hot‑spot architecture that keeps the most frequently accessed vectors in an in‑memory HNSW graph while delegating the long tail to DiskANN. You’ll get:

• A refresher on the two dominant ANN families (HNSW and DiskANN).
• Real‑world cost and performance numbers, with a transparent methodology.
• Step‑by‑step guidance for profiling, building, deploying, and monitoring a hybrid system.
• Operational best practices to survive traffic spikes, hardware failures, and evolving query distributions.

By the end of this guide you should be able to cut RAM spend by 40‑60 % while preserving sub‑3 ms latency for the vast majority of user‑facing queries.

2. The Real Cost of RAM‑Heavy Vector Search

2.1 Why RAM is the Bottleneck

• Dimensionality matters – Each 128‑dimensional float32 vector occupies 512 bytes. A dataset of 200 M vectors therefore needs ~100 GB just to store the raw embeddings.
• Graph overhead – HNSW stores adjacency lists for each node. Empirical studies (e.g., Malkov & Yashunin, 2020) show a typical memory factor of 1.5 ×–2 × the raw size, depending on M (the number of bi‑directional links per layer) and efConstruction.
• Memory pricing trend – According to AWS pricing history, the per‑GiB price for r5 memory‑optimized instances rose from $0.12/GB‑month in 2022 to $0.20/GB‑month in 2025, a ≈30 % YoY increase.

Putting the pieces together, a 200 M‑vector HNSW index on a 2025‑generation instance (e.g., r5.12xlarge) consumes roughly 250 GB RAM, costing ≈$4,500/month for the VM alone (compute + memory). Adding a second node for HA doubles that number.

2.2 The Business Impact

When a latency‑critical service cannot meet its <2 ms SLA, revenue loss can far outweigh the RAM bill. Overspending on memory for a non‑critical batch job is wasteful.

Recent incidents illustrate the stakes:

Incident	Trigger	Failure Mode	Lesson for Vector Search
PlayStation Network login outage (Eurogamer, Mar 2026)	Sudden beta‑tester surge for a new game	Authentication service OOM‑killed	RAM‑only services can collapse under flash‑crowds
NASA Deep Space Network loss (New Scientist, Jan 2026)	Wildfire took out a ground station	Loss of telemetry path	Single‑point hardware failures cripple high‑availability systems
FinTech fraud‑detection latency breach (internal case study, Q2 2025)	Market volatility spiked transaction volume 8×	HNSW node hit memory bandwidth saturation → 5 ms latency	Memory bandwidth, not just capacity, is a limiting factor

These examples converge on a single point: design for graceful degradation. When RAM is exhausted, the service should not crash; it should fall back to a slower but still functional path.

3. Understanding ANN Index Fundamentals

3.1 Approximate Nearest Neighbor (ANN) Overview

ANN algorithms accelerate the k‑nearest neighbor (k‑NN) problem by building a data structure that prunes the search space. The two most widely adopted families for high‑dimensional dense vectors are:

Family	Core Idea	Typical Memory Footprint	Typical Latency (large dataset)
Hierarchical Navigable Small World (HNSW)	Multi‑layer proximity graph; greedy search from top layer down	1.5 ×–2 × raw vectors (full adjacency)	<1 ms (RAM‑resident)
Disk‑Optimized ANN (DiskANN)	Shallow in‑memory entry graph + deep adjacency stored on SSD; uses OS page cache	0.1 ×–0.15 × raw vectors (entry graph + cache)	10‑30 ms (NVMe‑resident)

Both assume a static dataset after bulk loading. Incremental updates are possible (e.g., HNSW add_point), but they incur re‑balancing costs and can degrade recall if not re‑indexed periodically.

3.2 HNSW in Detail

• Construction – Vectors are inserted one by one. Each insertion walks the current graph to find a suitable insertion point, then creates links to up to M nearest neighbors in each layer.
• Search – Starts at the top layer (few nodes) and performs a greedy descent, followed by a best‑first search in the bottom layer limited by efSearch. The algorithm is logarithmic in N for well‑tuned parameters.
• Parameters – M (graph degree) improves recall but increases memory. efConstruction controls construction effort; larger values give higher quality graphs. efSearch trades recall for runtime; larger values increase recall at the cost of more distance computations. Typical production settings: M=32, efConstruction=200, efSearch=64.

3.3 DiskANN in Detail

DiskANN augments HNSW with a two‑level storage layout:

1. Entry Graph (RAM) – A shallow HNSW (often M=16, efConstruction=100) that fits comfortably in a few gigabytes.
2. Adjacency Graph (SSD) – Full neighbor lists stored in a custom binary format, compressed with product quantization (PQ) or scalar quantization to reduce I/O volume.

During a query:

• The entry graph quickly narrows the search to a set of candidate nodes.
• The algorithm then streams deeper adjacency blocks from SSD, using OS page cache to keep hot blocks in memory.
• Modern NVMe drives provide ~3 GB/s sequential read, bounding the I/O cost per query.

Key tuning knobs:

Parameter	Effect
max_degree (on‑disk)	Controls how many neighbors are stored per node; higher values improve recall but increase index size.
search_width	Number of entry‑graph candidates expanded before hitting disk; larger values reduce latency but increase SSD load.
cache_size_gb	Amount of RAM reserved for the SSD page cache; typical 8‑16 GB for a 1‑2 TB index.

4. In‑Memory HNSW: Performance at a Premium

4.1 Benchmark Snapshot

Metric	c5.9xlarge (36 vCPU, 72 GB)	r5.12xlarge (48 vCPU, 384 GB)
Dataset	200 M × 128‑dim float32	Same
RAM usage	250 GB (incl. OS)	250 GB
99th‑pct latency	0.9 ms	0.8 ms
Throughput (QPS)	12 k	13 k
Monthly cost (on‑demand)	$4,500	$5,200
HA (2× nodes)	$9,000	$10,400

All numbers are measured with faiss‑compatible HNSW (M=32, efSearch=64) on a warm cache.

4.2 When HNSW Is the Right Choice

Scenario	Latency Requirement	Data Size	Update Frequency	Cost Tolerance
Real‑time product recommendation during checkout	<2 ms	≤ 500 M vectors	Daily bulk re‑index	High
Voice‑assistant intent matching	<1 ms	≤ 100 M vectors	Hourly incremental adds	High
Fraud detection in high‑frequency trading	<5 ms (soft)	≤ 200 M vectors	Near‑real‑time inserts	Medium‑High

If sub‑millisecond latency is a hard SLA, HNSW is still the only proven solution at scale.

4.3 Cost Drivers & Trade‑offs

Driver	Impact	Mitigation
Memory price	Directly scales with index size	Use RAM‑optimized instances, negotiate reserved capacity discounts
GC pauses (Java) / memory fragmentation (C++)	Can add 0.5‑2 ms jitter	Tune JVM heap, use off‑heap libraries (e.g., faiss C++ bindings)
Memory bandwidth saturation	Limits QPS under flash‑crowd	Deploy multiple shards, enable NUMA‑aware placement
Hot‑spot skew	A few vectors dominate traffic, causing cache thrashing	Move hot vectors to a dedicated HNSW shard (see hybrid)

5. DiskANN: Scaling Cost‑Effectively

5.1 Benchmark Snapshot

Metric	i3en.metal (96 vCPU, 384 GB, 8 TB NVMe)
Dataset	200 M × 128‑dim float32
RAM usage	32 GB (entry graph + 8 GB cache)
Disk usage	1.2 TB (compressed adjacency)
99th‑pct latency	14 ms
Throughput (QPS)	3 k (single node)
Monthly cost (on‑demand)	$1,600 (compute + 8 TB SSD)
HA (2× nodes + RAID‑10)	$3,500

All numbers are measured with Microsoft’s diskann library (max_degree=64, search_width=32, cache_size_gb=12).

5.2 When DiskANN Is the Right Choice

Use case	Latency	Data Size	Refresh Frequency	Cost Tolerance
Offline batch similarity generation (nightly)	≤30 ms	> 1 B vectors	Weekly bulk load	Low‑to‑Medium
Content‑based image retrieval for a news portal (user‑facing but tolerant)	≤20 ms	500 M vectors	Monthly refresh	Medium
Embedding‑as‑a‑service for internal tooling	≤15 ms	300 M vectors	Daily bulk load	Low‑Medium

If 10‑20 ms latency is acceptable, DiskANN can slash RAM spend by ≈70 %.

5.3 Practical Tips for DiskANN Deployment

1. SSD Selection – Choose NVMe drives with ≥3 GB/s sequential read and low write latency (e.g., Intel Optane 900P). Avoid SATA SSDs.
2. File System – Use xfs or ext4 with noatime and discard=async.
3. Page Cache Warm‑up – After a node restart, run a pre‑warm script that issues a low‑rate query for each entry‑graph node to pull hot adjacency blocks into RAM.
4. RAID Configuration – For HA, RAID‑10 offers a good balance of read performance and fault tolerance.
5. Monitoring I/O – Track iostat metrics (await, svctm, %util). If %util exceeds 70 % consistently, consider adding another DiskANN node or increasing search_width.

6. Hybrid Architecture: Best of Both Worlds

6.1 Design Overview

+-------------------+          +-------------------+
|   Hot‑Spot HNSW   |  <----> |   Routing Service |
^                               |
|                               v
|   Cold‑Tail DiskANN|  <----> |   Cold‑Tail Nodes |

• Hot‑Spot HNSW holds the N most‑queried vectors (typically 5‑10 % of the dataset).
• Cold‑Tail DiskANN stores the remaining 90‑95 % on SSD.
• Routing Service decides, per query, whether to hit the hot or cold tier.

6.2 Determining the Hot Set

1. Collect Query Logs – Store each query’s vector ID (or hash) with a timestamp in a time‑series store (e.g., ClickHouse, Amazon Timestream).
2. Compute Frequency – Run a Spark job that aggregates counts over a sliding window (e.g., last 24 h).
3. Select Threshold – Choose a percentile (e.g., top 8 % of IDs) or a fixed count (e.g., 15 M vectors) that fits within the RAM budget of your hot node.

Sample Spark code (Scala):

val logs = spark.read.parquet("s3://my-bucket/query-logs/")
val hotIds = logs
  .groupBy($"vector_id")
  .agg(count("*").as("freq"))
  .orderBy(desc("freq"))
  .limit(15_000_000)
  .select("vector_id")
  .as[Long]
  .collect()
  .toSet
Persist the resulting ID set to a distributed KV store (e.g., DynamoDB) for the routing service to query in O(1) time.

6.3 Building the Two Indexes

Step	Action	Command (FAISS)
1	Export hot vectors to hot_vectors.npy	python export_hot.py --ids hot_ids.txt --out hot_vectors.npy
2	Build HNSW on hot set	faiss.IndexHNSWFlat(d, 32).train(hot_vectors.npy)
3	Export cold vectors (remaining)	python export_cold.py --exclude hot_ids.txt --out cold_vectors.npy
4	Build DiskANN on cold set	diskann_build --input cold_vectors.npy --out diskann_cold.index --max_degree 64 --search_width 32 --cache_size_gb 12

Both indexes use the same distance metric to avoid mismatches at the routing layer.

6.4 Routing Logic

A minimal routing microservice (Python + FastAPI) could look like:

from fastapi import FastAPI
import numpy as np
import faiss
import diskann

app = FastAPI()

# Load hot ID set (Bloom filter for memory efficiency)
hot_filter = BloomFilter(max_elements=15_000_000, error_rate=0.001)
hot_filter.load("hot_filter.bloom")

# Load in‑memory HNSW index
hnsw = faiss.read_index("hnsw_hot.bin")

# Load DiskANN client (wraps the native library)
disk = diskann.DiskANN("diskann_cold.index")

@app.post("/search")
async def search(vec: List[float], k: int = 10):
    query = np.array([vec], dtype='float32')
    if hot_filter.might_contain(hash(query.tobytes())):
        D, I = hnsw.search(query, k)
        source = "hot"
    else:
        D, I = disk.search(query, k)
        source = "cold"
    return {"distances": D.tolist(), "ids": I.tolist(), "source": source}

The Bloom filter keeps the routing decision to a single memory read and hash operation, adding under 0.1 ms.

6.5 Scaling the Hot Tier

Because the hot tier is RAM‑intensive, you may need multiple shards to meet QPS targets. A typical pattern:

• Shard by hash range – Split the hot ID space into S shards (e.g., 4 shards each holding ~3.75 M vectors).
• Load balancer – Use a layer‑7 LB (Envoy, NGINX) that forwards queries based on the Bloom filter result and a consistent‑hash of the vector ID.
• Auto‑scale – Deploy each shard as a Kubernetes Deployment with an HPA that watches custom latency metrics (search_latency_ms).

6.6 Fail‑over Path

If a hot node crashes:

1. The routing service detects the missing health check and flips a flag.
2. All queries are sent to DiskANN with a fallback header (X-Search-Mode: cold).
3. Latency increases (e.g., from 0.9 ms to 14 ms) but the service remains available.

When the hot node recovers, the routing service re‑populates the Bloom filter and resumes normal operation.

6.7 Real‑World Numbers

A media‑streaming platform (≈ 300 M vectors, 128‑dim) implemented the hybrid design:

Metric	Before (pure HNSW)	After (Hybrid)
RAM usage	250 GB (single node)	80 GB (hot shards) + 32 GB (cold) = 112 GB
Monthly cost	$5,200 (2× HA)	$2,800 (hot HA + cold single)
99th‑pct latency	0.9 ms	2.4 ms for 95 % of queries, 14 ms for 5 %
QPS (peak)	12 k	13 k (hot) + 3 k (cold) = 16 k
Hot‑spot share	N/A	8 % of vectors, 95 % of traffic

The cost reduction came primarily from shrinking RAM, while the latency SLA (<3 ms for user‑facing calls) stayed intact because the hot‑spot covered the overwhelming majority of traffic.

7. Operational Resilience: Lessons from Outages

7.1 Traffic Spikes & Flash‑Crowds

A sudden surge can increase query volume 5‑10× within minutes. In a pure‑RAM deployment, this can exhaust memory bandwidth, causing CPU stalls, GC pauses, or OOM kills.

Mitigation in Hybrid Setup

Technique	How It Helps
Latency‑based circuit breaker	If latency > 5 ms, automatically route all traffic to DiskANN, buying time for hot nodes to scale.
Custom HPA metric	Scale hot shards based on search_latency_ms rather than CPU alone.
Hot‑spot re‑balancing	Re‑run the hot‑set computation every hour; newly popular vectors are promoted to the hot tier.

7.2 Hardware Failures

Disk failures are more likely on high‑density NVMe arrays. The hybrid model isolates the risk:

• Cold tier – Use RAID‑10 and replicate the DiskANN index across two availability zones.
• Hot tier – Keep at least two replicas of each shard (active‑active) and use a consensus protocol (Raft) to keep the in‑memory graphs synchronized.

If a cold node goes down, the routing service can rewire queries to the surviving replica without impacting the hot tier. If a hot node fails, the fallback to DiskANN ensures continuity, albeit with higher latency.

7.3 Monitoring & Alerting

Metric	Recommended Tool	Alert Threshold
Hot‑tier RAM usage	Prometheus node_memory_Active_bytes	> 85 % of allocated
Cold‑tier SSD I/O latency (await)	iostat exported to Prometheus	> 5 ms
Search latency (p99)	OpenTelemetry + Grafana	Hot tier > 2 ms, Cold tier > 20 ms
Hot‑spot share	Custom Spark job output	> 12 % of total queries (drift)
Health checks	Kubernetes liveness/readiness probes	Fail after 3 consecutive misses

Automated remediation (e.g., Kubernetes Jobs that rebuild the hot index) can be triggered from these alerts.

8. Future‑Proofing: Adapting to Shifting Workloads

8.1 Dynamic Hot‑Spot Re‑partitioning

The distribution of queries rarely stays static. A meme, news event, or product launch can push a previously cold vector into the hot set overnight. To keep the hybrid architecture efficient:

1. Continuous Frequency Capture – Use a streaming platform (Kafka + ksqlDB) to maintain a real‑time count per vector ID.
2. Sliding‑Window Top‑K – Every hour compute the top‑K IDs and compare with the current hot set.
3. Graceful Promotion/Demotion – Promote new hot IDs to the Bloom filter and rebuild the hot index incrementally. Demote stale IDs back to DiskANN via a background re‑index job.

Pseudo‑code for incremental promotion:

new_hot_vectors = fetch_vectors(new_ids)
hnsw.add_with_ids(new_hot_vectors, new_ids)   # O(1) per vector
bloom_filter.add_many(new_ids)

8.2 Emerging Hardware Trends

• Persistent Memory (Intel Optane DC) – Near‑RAM latency with larger capacities (256 GB‑1 TB) at lower cost. DiskANN can be adapted to store the adjacency graph on PMEM instead of SSD, reducing latency to ~5 ms while keeping RAM low.
• GPU‑accelerated ANN – Libraries like faiss-gpu can run HNSW on GPUs, increasing throughput for batch workloads. However, GPU memory is even more expensive, so the hybrid model still makes sense for latency‑critical online traffic.

8.3 Algorithmic Evolutions

Research continues on graph‑based indexes with learned routing (e.g., Learned HNSW, GraphVite). These approaches can reduce the degree M while preserving recall, directly translating into lower memory usage. When such a library reaches production‑grade stability, you can swap the hot tier implementation without changing the overall architecture.

9. What This Actually Means

• Static, one‑size‑fits‑all indexing is dead. Cloud economics of 2025‑2026 make pure RAM solutions prohibitively expensive for large datasets.
• Hybrid hot‑spot architectures deliver sub‑3 ms latency for most queries, a 40‑60 % reduction in RAM spend, and built‑in resilience to traffic spikes and hardware failures.
• Operational discipline matters: regular hot‑spot profiling, automated index rebuilds, and latency‑aware autoscaling keep the system performant.

If you continue to rely on a single HNSW node to meet a sub‑millisecond SLA, you risk OOM crashes during flash‑crowds—exactly what happened to the PlayStation Network authentication service. By adopting the hybrid pattern today, you position your service to survive the next surge, whether it’s a viral game launch or a sudden increase in AI‑driven search traffic.

10. Step‑by‑Step Migration Plan

Phase	Goal	Action Items
0 – Baseline	Capture current cost & performance	• Run faiss HNSW benchmark on production dataset. • Record RAM usage, QPS, latency, monthly bill.
1 – Data Profiling	Identify hot vectors	• Export query logs to a data lake. • Run Spark job to compute top‑K IDs (target ≤ 10 % of total).
2 – Index Construction	Build hot & cold indexes	• Export hot vectors → hnsw_hot.bin. • Export remaining vectors → diskann_cold.index. • Store hot ID set in a Bloom filter (persist to S3).
3 – Service Refactor	Add routing layer	• Implement a lightweight gRPC/HTTP router. • Integrate Bloom filter check. • Deploy hot and cold services on separate node pools.
4 – Validation	Verify correctness & latency	• Run A/B test: 10 % traffic to hybrid, 90 % to existing. • Compare recall (≥ 95 % of baseline) and latency (≤ 3 ms for hot).
5 – Cutover	Switch production traffic	• Gradually increase traffic share to hybrid. • Decommission pure HNSW node after 48 h of stable operation.
6 – Automation	Enable continuous hot‑spot updates	• Deploy streaming job to recompute hot set hourly. • Trigger incremental hot index rebuilds via Kubernetes Jobs. • Set up alerts for hot‑share drift (> 12 %).
7 – Scaling & HA	Add redundancy	• Deploy at least two hot shards across AZs. • Enable RAID‑10 for DiskANN SSDs. • Configure circuit‑breaker fallback to cold tier.

11. Key Takeaways

• Profile first; knowing which vectors dominate traffic is the foundation of any cost‑saving strategy.
• Separate hot and cold tiers: keep the top 5‑10 % of vectors in an in‑memory HNSW; store the rest in DiskANN.
• Route intelligently with a Bloom filter or LRU cache to keep the decision cheap.
• Scale the hot tier with sharding and latency‑aware autoscaling; the cold tier can stay static and cost‑optimized.
• Monitor both memory and I/O; latency spikes often originate from either RAM pressure or SSD queue saturation.
• Refresh hot sets regularly; re‑compute hot‑set every hour or as fast as the domain requires.

12. Further Reading

• Best Practices for Scaling Vector Databases in Production
• Managing Latency Budgets in Hybrid ANN Architectures
• Cost‑Effective GPU Alternatives for Large‑Scale Embedding Generation

13. Conclusion

Vector search is no longer a niche component; it is a core performance‑critical service for many AI‑enabled products. The economics of cloud memory have shifted, making the classic “keep everything in RAM” approach untenable for large datasets. By splitting the workload into a hot in‑memory HNSW tier and a cold DiskANN tier, you can cut RAM spend by up to 60 % while still delivering sub‑3 ms latency for the overwhelming majority of queries.

Implement the hybrid pattern today, automate hot‑spot detection, and you’ll have a vector search service that is fast, affordable, and resilient—ready for the next wave of AI‑driven user experiences.

Read Next

• Best Way to Build Real-Time Pipelines for Space and Earth Observation Data
• How to Build a Scalable Pipeline for Space Science Data
• Innovative Methods in Time-Series Restoration and Neural Networks

Read next: continue with one of these related guides.

---

Originally published at The Looplet.