DEV Community: Ashish Mishra

Where keyword search and vector search each fail

Ashish Mishra — Tue, 08 Sep 2026 00:00:00 +0000

Sep 8, 2026 · engineering

We indexed 10,003 real questions that customers asked an online bank and ran the same queries through BM25 and vector search. The failures are instructive in both directions, and they are the whole case for hybrid search. This post shows the failures. The full walkthrough in the docs is runnable end to end: dataset, embeddings, index build, every query shape, and a read-back of the produced files with a plain Parquet reader.

The paraphrase that breaks keyword search

BM25 ranks rows by the exact tokens they share with the query. A user asking about a card that never arrived phrases it their own way, and BM25 anchors on "not shown up" and drifts into refund territory by the second hit:

keyword_miss.py

table.bm25_search("text", "my new card still has not shown up", 5)

17.39 Why has my new card still not come? 15.52 My refund has not shown up on my statement. 15.15 my check or cash deposit has not shown up 14.04 My refund has not shown up in my account. Where is it? 13.42 My statement has not shown my refund.

Vector search matches meaning, so the same paraphrase works even where a hit shares almost no vocabulary with the query:

semantic.py

table.vector_search("embedding", qvec, 5, projection=["text", "score"])

0.117 Why has my new card still not come? 0.165 why have i not got my new card? 0.172 I just got a new card how do I get it to start working? 0.180 where is my new card? 0.181 2 weeks ago I ordered my new card. It isn't here. What should I do?

The token that breaks vector search

Invert the query and the failure inverts with it. On a term of art, BM25 is precise:

keyword.py

table.bm25_search("text", "SWIFT transfer", 5, projection=["text", "score"])

9.46 Is a SWIFT transfer acceptable? 9.46 Can I transfer using SWIFT? 9.46 Can I Transfer from SWIFT? 9.08 Can I do a SWIFT transfer? 8.72 is there a fee for swift transfer

Vector search has no notion of an exact token. A rare identifier, a product name, or an error code can dissolve into its embedding neighborhood, outranked by rows that are merely on topic. A real query stream contains both shapes daily. Whichever single retriever you pick, some of your users lose.

Run both, fuse the rankings

Hybrid search runs both retrievers and merges the two ranked lists with reciprocal rank fusion, so a row both retrievers surface outranks a row only one of them found:

hybrid.py

q = "top up with Apple Pay"
table.hybrid_search("text", q, "embedding", model.encode(q).tolist(), 5,
                    projection=["text", "score"])

0.0323 Is top-up possible with Apple Pay? 0.0320 How can I top up with Apple Pay? 0.0308 Am I able to top up with Apple Pay? 0.0161 My Apple Pay top up with my American Express card is not working? 0.0161 is there a way to do top up with apple pay

The exact-token half of the query (Apple Pay) and the intent half (topping up) each pull their weight. Neither retriever alone ranks this list this way. If you want the concept in depth, what is hybrid search? covers the two retrievers and the fusion math.

All of this ran directly on Apache Parquet files. The index lives inside the files, they stay readable by any plain Parquet reader with Infino nowhere in the read path, and the same code targets S3 by changing one string. The docs guide is the complete walkthrough: install, dataset, embeddings, the index build, filtered search, SQL composition over search results, and the read-back. Inside a Parquet superfile covers how the format carries an index without breaking compatibility.

Fixed grids on vector search - comparing 3 approaches to quantization: FAISS, Turbovec, and Infino

Ashish Mishra — Tue, 01 Sep 2026 00:00:00 +0000

Sep 1, 2026 · engineering

We benchmarked three approaches to 4-bit quantized vector search over the same 100,000 OpenAI embeddings (1536 dimensions): FAISS's classic product quantization, turbovec — an open-source implementation of TurboQuant — and Infino's SQ4, which we build, each measured through its own public API against exact brute-force ground truth. At 4 bits per dimension every index stores 768 bytes of codes per vector, and all three sit between 0.94 and 0.97 recall — but the latencies run from 1.5 ms to 45 ms.

The sections below work through where that difference comes from, build and write cost, and the same scan at four corpus sizes.

Quantization, briefly

There are two ways to make approximate search cheaper, and they're independent of each other.

a. You can look at fewer vectors. That's routing : IVF (inverted file) groups the corpus into clusters and reads only the clusters nearest your query, HNSW (hierarchical navigable small world) walks a graph toward the answer, tree methods carve up the space.

b. Or you can make each vector cheaper to look at. That's quantization , and it's the only thing that varies here — every index in this comparison scans all 100,000 vectors on every query. The FAISS index string carries no coarse quantizer, the turbovec index is a flat scan by design, and the Infino mode we measured is an exhaustive scan over its in-memory codes.

Scalar quantization treats each coordinate on its own: take the float, round it to one of 16 levels, store 4 bits. 1536 coordinates, 768 bytes. Usually there's a rotation first, and those 16 levels can be evenly spaced or fitted to the data — Lloyd-Max, the classic minimum-error placement, is one such fit. Product quantization chops the vector into M pieces and runs k-means separately on each piece, replacing that chunk of the vector with the index of its nearest centroid. The list of centroids a code indexes into is called a codebook, and each piece's codebook is a subquantizer. Because the centroids live in the chunk's own space, they can capture correlation between coordinates in a chunk, which per-coordinate rounding cannot. The price is a training pass over the corpus.

One thing none of these indexes does: rerank. In many systems you take the top few hundred results then re-score those against the uncompressed vectors.

The three approaches

FAISS is Meta's vector library, and its indexes are specified by factory string. The one here is IDMap,PQ768x8np — classic product quantization with a lookup table: 768 subquantizers of two dimensions each, so one 8-bit code covers two coordinates and the budget is the same 4 bits per dimension and 768 bytes per vector as everything else in the article. It is also the configuration turbovec's own README benchmarks its recall against.

turbovec implements TurboQuant, and we ran it at its 1.0.0 release. No routing structure at all, a flat scan over compressed vectors. Per its docs, each vector is rotated, quantized to 2 or 4 bits per coordinate against a Lloyd-Max codebook, and renormalized by length, and the search kernels use nibble-split lookup tables over an interleaved layout, chosen at runtime.

The TurboQuant authors' own released code ships only a conceptual Python path for inner-product estimation — no optimized scan, which is why the RaBitQ team's comparison report excluded query-time efficiency entirely. As far as we know turbovec is the only way to measure this method's scan at full speed.

Infino is an embedded retrieval library whose tables are Parquet files with the vector index embedded in them. It doesn't implement one fixed vector algorithm. The index serves through one of several modes — a routed cluster scan by default, an in-memory graph, or a flat scan — picked by a config setting, and optimize() sizes whichever is selected against a recall target measured on the table's own data. The mode tested here is the flat scan, flat_ivf: a 4-bit plane scanned exhaustively. At small corpora it's the effective shape and it's the configuration comparable to a flat library scan.

The setup

Everything runs in one process over 100,000 vectors from the dbpedia OpenAI corpus — 1536 dimensions, cosine — each engine through its own public API. The machine is an EPYC 9V74 slice: 4 cores, 8 hardware threads, 63 GiB of RAM.

Resident memory at 100,000 vectors: fp32 against the 4-bit index.

At fp32, this corpus is 586 MiB of vectors. Every index below serves it from about 75 MiB — 4 bits per dimension in place of 32. Quantization writeups often make this the headline — turbovec's README included, where the compression figures ("31 GB of RAM as float32… fits it in 4 GB", "16x compression") are quoted against full 32-bit floating point vectors. The reduction in memory is real, but it is also what any 2- or 4-bit quantization delivers.

To measure recall, ground truth is exact brute-force nearest neighbors, computed once and reused. Recall is graded at k = 1, 10 and 100 over 1,000 held-out queries, and every engine answers the same query set.

Latency, memory and recall

Queries run one at a time, so the latency measurements are per-query scan costs and say nothing about throughput. Nothing pins threads: each engine parallelizes a single query however it chooses.

| | recall@10 | p50 | bytes/vector | resident |
| --- | --- | --- | --- | --- |
| FAISS flat PQ | 0.950 | 45.5 ms | 791 | 75.5 MiB |
| turbovec 4-bit | 0.948 | 1.59 ms | 788 | 75.2 MiB |
| Infino 4-bit | 0.963 | 1.50 ms | 788 | 75.2 MiB |

Infino is slightly faster than turbovec, and both are significantly faster than FAISS.

As expected, memory differences are a wash. 1536 dimensions at 4 bits is 768 bytes of codes, and all three indexes land within 3% of that, at 788 to 791 bytes per vector.

The rest is bookkeeping. turbovec and Infino each store 20 bytes: an id plus a per-vector scalar that corrects the quantized dot's bias. Flat PQ stores an 8-byte id plus its trained codebook, which is 1.5 MiB shared across 100,000 vectors, or 15.7 bytes each — 768 + 8 + 15.7 = 791.7, against 791 measured.

Recall at k = 1, 10, 100:

| | k=1 | k=10 | k=100 |
| --- | --- | --- | --- |
| FAISS flat PQ | 0.949 | 0.950 | 0.963 |
| turbovec 4-bit | 0.946 | 0.948 | 0.957 |
| Infino 4-bit | 0.970 | 0.963 | 0.970 |

Every number here is graded over 1,000 held-out queries, which puts the standard error on a mean recall near ±0.004. FAISS PQ and turbovec are within a couple of points of each other at every k. Infino leads the group by at least 1.3 to 1.8 points at every size we ran.

Searching the vector space

A 4-bit code can take sixteen values, so every quantizer here has sixteen levels to place. The three approaches place them differently.

What each approach trains, stores, and scores with.

FAISS trains centroids. It runs k-means in 768 two-dimensional subspaces, 256 centroids each — that's the 312 CPU-seconds of build time, and it buys a codebook that can represent correlation between paired coordinates, which neither one-dimensional fit in this comparison can. The scan never compares the query against the compressed vectors directly: once per query it precomputes the distance to every centroid — a 768 KiB table, 768 × 256 floats — then scores a candidate by summing one lookup per subquantizer, 768 of them, each at an address not known until the code byte has been read.

Turbovec precomputes its levels. The insight behind it is not TurboQuant's alone — rotate-then-quantize is a family: RaBitQ (SIGMOD 2024) established the construction and its error bounds although there is an ongoing discussion on precedence. The property both rely on is the same: after a random rotation, every coordinate follows the same known distribution — a Beta, near-Gaussian in high dimension — so the optimal 16-level Lloyd-Max quantizer for that distribution can be computed once, ahead of time, and reused for every corpus forever. (Infino's ruler, next, is the same construction with uniform levels.) Nothing is trained and nothing is fitted; the 1.2 CPU-seconds is rotating and encoding, which is the point — the paper calls it data-oblivious: no pass over the data is ever needed, so a vector can be quantized the moment it arrives. turbovec also stores one scalar per vector that corrects the inner product's quantization bias. At scan time the fitted levels are read through lookup tables of 16 entries per coordinate — small enough to stay in SIMD registers — at 1.59 ms.

Infino computes a ruler based on the data. It rotates (a seeded structured rotation — only the 8-byte seed is persisted), then fits offset and step per rotated coordinate over mean ± 2.7σ , uniform steps between. No codebook at all. The 2.7 is the optimal 16-level loading for a Gaussian, and the rotation is what makes the Gaussian assumption hold: it spreads each coordinate toward the same near-Gaussian marginal, which is what lets one global ruler — a fixed grid, the same sixteen levels for every row in the table — work at all; on the raw embedding axes, with their wildly uneven energy, most coordinates would collapse onto a couple of codes. A comment in the encoder records the measured stakes: loading over sigma instead of min/max moved recall@10 up 0.032 on this corpus, because a min/max ruler is set by each coordinate's single most extreme value and spends most of its 16 levels on range almost no row occupies. The scan carries no distance table of any size — because the levels are evenly spaced, scoring reduces to a direct integer dot on the packed nibbles — and runs at 1.50 ms.

Building the vectors

| | wall (8 threads) | CPU-seconds |
| --- | --- | --- |
| turbovec 4-bit | 374 ms | 1.2 |
| Infino¹ | 1.7 s | — |
| FAISS flat PQ | 39.7 s | 312 |

¹ Infino's build is a table write: the vectors are appended and committed, so the 1.7 s ends with the data durable on disk as Parquet, where the library builds end in RAM. With a single writer it's 3.57 s. The harness doesn't record CPU-seconds for it.

FAISS trains: 768 k-means problems, 256 centroids each over a two-dimensional subspace, iterated over a sample of the corpus — that's the 312 CPU-seconds. TurboQuant's codebook is precomputed for the rotated coordinate distribution and never reads the corpus; turbovec's 1.2 CPU-seconds is rotation and encoding. Infino's build trains too — the 1.7 s includes its own k-means — but we didn't profile it by stage.

On this corpus the trained codebook and the precomputed one reach the same recall at the same bytes. That's corpus-dependent, and we ran only the one. We didn't test anything with strong subspace structure, which is exactly where trained codebooks would have more to fit than a per-coordinate scheme does.

A new vector has two jobs: get into the search structure, and get onto disk. FAISS and turbovec split them. Their add writes straight into the layout the scan kernels read, so after turbovec's 18 µs the vector is searchable — in RAM, where a crash takes it back, because nothing touches disk until save rewrites the whole index. Infino's append does both jobs in one call: when it returns, the rows are searchable and committed.

| | add | remove | persist | full cycle |
| --- | --- | --- | --- | --- |
| FAISS flat PQ | 3.14 ms | 401 µs | 15.2 ms | 68.5 ms |
| turbovec 4-bit | 18.0 µs | 3.4 µs | 33.4 ms | 21.4 ms |
| Infino 4-bit | 27 µs¹ | 2.7 µs¹ | — | 17.0 ms |

¹ per row, 100,000 rows sharing one commit, for the append and the delete alike. Infino separates add and save operations internally but renders them combined in its public API: append and delete each do the in-memory work and the save in one call, so the per-row cost is set by how many rows share it. A row alone in the commit costs the full 17 ms to add, 7.9 ms to delete.

Deletes have the same latency shape — turbovec's 3 µs remove is the in-RAM half, durable on its next save, and Infino's is again a commit — with one twist: a tombstone — the committed delete marker — carries no vector payload, so a wide delete amortizes below the in-RAM number, 2.7 µs a row durable against 3.4 µs in RAM. The two write models come down to what gets batched: the libraries batch time, with mutations waiting in RAM until the next save and a crash losing the interval, while Infino batches rows, and every call is a durability boundary.

Committing every write costs the scan nothing: 788 bytes per vector and 1.50 ms, inside the range of the two libraries.

Scaling the quantization

Everything above shares one constraint: to answer a query, each of these indexes reads every row. A SIMD scan over contiguous bytes has the DRAM and L1/L2/L3 cache behavior that random pointers through data structures can't match, whether it is a graph, a tree, or other types of routing. At small corpora, flat scan is the natural shape.

We ran the same benchmark at four corpus sizes:

| warm p50 @ k=10 | 100K | 250K | 500K | 933K |
| --- | --- | --- | --- | --- |
| Infino SQ4 | 1.50 ms | 4.64 ms | 9.90 ms | 18.88 ms |
| turbovec 4-bit | 1.59 ms | 4.84 ms | 10.19 ms | 19.05 ms |
| FAISS flat PQ | 45.5 ms | 111.8 ms | 226.3 ms | 414.7 ms |

Recall over the same sweep:

| recall@10 | 100K | 250K | 500K | 933K |
| --- | --- | --- | --- | --- |
| Infino SQ4 | 0.963 | 0.962 | 0.960 | 0.960 |
| turbovec 4-bit | 0.948 | 0.945 | 0.941 | 0.937 |
| FAISS flat PQ | 0.950 | 0.945 | 0.942 | 0.942 |

Infino recall holds within 0.3 points across the sweep. turbovec and flat PQ drift down about a point. However, given the linear nature of scans all 3 libraries incur latency penalties as the corpus scales. In the next post we'll show how we scale past that point, with different ANN (approximate nearest neighbor) structures built on the same fixed-grid philosophy.

Limits

FAISS also ships a SIMD variant of this index: PQ1536x4fs, "FastScan," which trades the float lookup tables for 8-bit-quantized ones resolved by shuffle instructions. We measured it, and its speed is real — 4.4 ms on this corpus, 10× faster than the classic scanner. Its recall is not publishable: at this 1536-subquantizer geometry it swung between 0.59 and 0.91 across corpus sizes in our runs, and a control that scored the identical trained codebook through the classic float tables read a steady 0.934 — so the loss is inside the fast-scan kernel, not the quantization. We found no published comparison that reports FastScan recall at all; turbovec's own README uses classic PQ as its recall baseline and cites FastScan only as a speed reference, and FAISS normally deploys it with a reranking stage.
FAISS PQ is normally deployed at 32 to 96 bytes per vector, where its lookup table would be 32 to 96 KiB; the 768 bytes per vector measured here is far outside its usual range.
Queries run one at a time throughout, so nothing in the test measures concurrency.
Every recall number is a raw quantized operating point; we didn't measure any of these indexes with a rerank stage.

Reproducing it

reproduce.sh

git clone https://github.com/infino-ai/retrievalbench && cd retrievalbench
printf 'vector:\n search_mode: flat_ivf\n' > infino.yaml
INFINO_BENCH_SUPERTABLE_DOCS=100000 \
  cargo bench -- vector-codec \
  corpus=hf:KShivendu/dbpedia-entities-openai-1M corpus-dir=./corpora

The scaling table is the same command at INFINO_BENCH_SUPERTABLE_DOCS of 250000, 500000, and 1000000; the recall and latency tables all come from that one four-size sweep, and the build and write tables from the same harness at 100K. The corpus downloads once from HuggingFace. The FAISS row needs --features faiss after scripts/build_faiss.sh, which builds the bundled FAISS source with -march=native — without it, FAISS's SIMD kernels silently fall back to scalar code. The engine dependencies are pinned to a commit, and every committed run records host, engine commit and command in a run.json beside the numbers under results/inprocess/.

What is hybrid search?

Ashish Mishra — Tue, 25 Aug 2026 00:00:00 +0000

Aug 25, 2026 · engineering

Hybrid search runs two retrievers over one query. A keyword retriever scores documents with BM25 against an inverted index. A vector retriever finds nearest neighbors to an embedding of the query. Each returns its own ranked list, and a fusion step merges the two into one ranking.

The reason to run both is that they miss different things.

Two retrievers, opposite failure modes

BM25 is literal. It finds the error code, the part number, the surname, the exact phrase someone typed, and it misses the same idea written another way.

Vector search is about meaning. It finds the paraphrase, the synonym, the other language, and it will also hand back things that are merely on topic.

A query like retry a failed request needs both: the token retry, and the idea of backoff, try-again, transient failure.

A question in clauses

Take “Which services retried a failed request, by team?”. Each clause asks for something different.

Which services is a count: COUNT(DISTINCT service).
By team is a GROUP BY: one count per team.
Retried is a word to match. BM25 finds the events that say it.
A failed request has no fixed wording. Vector search finds “try again”, “transient failure”, “exponential backoff”.

Fusing two incompatible scores

BM25 similarity rises for a better match, while vector distance falls. The two numbers are not on the same scale and adding them means inventing a weight. Reciprocal rank fusion sidesteps that by using position in each list instead of the scores themselves.

rrf.txt

contribution(rank) = 1 / (60 + rank)

# first in a list contributes 1/61
# a document first on both sides: 1/61 + 1/61 ≈ 0.0328
# first on one side only: 1/61 ≈ 0.0164
# tenth on both sides: 1/70 + 1/70 ≈ 0.0286

Each list is best-first. A hit at rank r (first place is 1) contributes 1 / (60 + r). The two contributions add, and the sum is the emitted score. A document both retrievers liked climbs above a document that only one of them ranked first.

60 is the constant from the 2009 paper that introduced the method, where fusing runs this way beat every individual system and Condorcet Fuse. It keeps fusion deterministic across queries.

Fusion needs a stable identity to merge on. A row's _id is what makes the same document one document, even when the two indexes store it differently.

k controls recall

Each lane retrieves k hits and fusion keeps the top k by fused score. A result LIMIT is a separate number: how many of those rows the query returns.

A result page uses a modest k (50-200) and a small LIMIT.
An aggregate over matching events uses a large k (thousands) so GROUP BY and COUNT see the full candidate cohort.
The disagreement set (rows only one retriever found) also needs a deep k. Shallow k hides the misses.
Raising k costs work on both lanes and increases retrieval depth without changing fusion weights.

Seeing where the two disagree

Fusion hides the split by design. To measure it, run each retriever on its own and join the two relations on _id. The rows on only one side are the ones a single retriever never returns.

disagreement.sql

-- where do keyword and meaning disagree, and how much does each add

WITH sides AS (
  SELECT CASE
           WHEN v._id IS NULL THEN 'keyword only' -- literal, no paraphrase
           WHEN k._id IS NULL THEN 'meaning only' -- the same thing, said differently
           ELSE 'both agreed'
         END AS found_by,
         k.score AS bm25,
         v.score AS cosine
  FROM bm25_search('tickets', 'body', 'disk full on ingest', 100) k
  FULL OUTER JOIN
         vector_search('tickets', 'embedding', :q, 100) v
      ON v._id = k._id -- one snapshot, both sides
)
SELECT found_by,
         count(*) AS docs,
         round(avg(bm25), 3) AS avg_bm25, -- nulls skip themselves
         round(avg(cosine), 3) AS avg_cosine
FROM sides
GROUP BY found_by
ORDER BY docs DESC;

Run it per query class. When the “meaning only” bucket grows, the embeddings and the vocabulary are drifting apart. Note that k.score is BM25 (higher is better) while v.score is a distance (lower is better), which is the reason fusion compares rank positions rather than adding those columns.

What sits around retrieval

Hybrid search is the retrieval middle: one keyword query, one vector query, fusion, a ranked list. The quality stages on either side of it belong to the application.

Query rewrite sits in front. Several rewritten queries can run as separate searches or a UNION.
A cross-encoder reranker sits after, on the dozens of rows that came back. Rescoring candidates inside an ANN index is a different thing that shares the name.
Chunking, contextual retrieval, and a labeled eval set are ingest and evaluation work.

In Infino, both indexes live inside the same Parquet files, both lanes read one pinned snapshot, and the fused result is a SQL relation: hybrid search on Infino · the query surface.

Why object-storage vectors use OPANN + Sq16, while RAM uses HNSW

Ashish Mishra — Thu, 20 Aug 2026 00:00:00 +0000

Aug 20, 2026 · engineering

HNSW is an excellent in-memory vector index. It is also a poor object-storage access pattern. Each graph hop depends on the previous hop, so a cold walk turns pointer chasing into serialized range requests.

Infino therefore uses two serving paths over the same vector data: HNSW when the working set is resident in RAM, and OPANN (Infino’s object-storage ANN path) with Sq16 when the vectors are served from Parquet on object storage.

The same corpus has two hardware states

A 1024-dimension fp32 vector occupies 4 KiB before index metadata. Ten million vectors are roughly 40 GiB of raw values. If that working set and its graph fit in the available RAM, spending memory to minimize query CPU and latency is reasonable.

The same decision fails for a large or mostly cold corpus. Keeping every vector and graph edge resident makes storage cost scale with the full dataset, even when only a small fraction receives queries. Moving the graph itself to object storage avoids the RAM bill but replaces memory loads with dependent network reads.

two-serving-states.txt

hot working set
  query ─▶ HNSW graph walk ─▶ top-k
          random access in RAM

cold / object-storage working set
  query ─▶ OPANN routing ─▶ bounded cluster ranges ─▶ top-k
                           contiguous object reads

durable source for both
  parquet superfiles in object storage

HNSW: random access is a feature in RAM

HNSW builds a navigable graph over the vectors. A query starts at an upper layer, follows promising neighbors toward the query, and expands a candidate set near the bottom. The graph avoids scoring most vectors.

In RAM, those dependent hops are cheap. The next node address may be unpredictable, but it resolves as a memory access rather than a storage request. A graph over the resident Sq16 representation also reduces the bytes touched per score relative to fp32.

The cost moves to build time and memory. The graph stores edges in addition to vector data, and loading or rebuilding it is work a cold worker must pay before it receives the low-latency path. HNSW is the right answer only when the working set is actually resident.

Why graph hops do not belong on object storage

Object storage rewards planned reads. One request can fetch a useful contiguous range at high throughput, but every additional dependent request pays request latency again.

An HNSW walk cannot issue all of its reads in advance: the node selected at step N determines which neighbors exist at step N+1. Even with caching, a cold graph walk creates the exact sequence object storage handles poorly: small, unpredictable, serialized reads.

Downloading the whole graph before searching avoids that request pattern, but turns the first query into a bulk load and requires enough local capacity for every cold namespace. That is pinning under another name.

OPANN: route first, then read contiguous ranges

The object-storage path groups vectors into regions that can be addressed independently. A small routing structure chooses the regions most likely to contain the nearest results. The query then fetches those cluster ranges, scores their compact vectors, and reranks a shortlist.

The important property is that the expensive reads are known before the data arrives. Cluster ranges can be requested concurrently, coalesced, cached, and reused. A query touches a bounded subset of the vector region instead of traversing a graph one network-dependent node at a time.

Because the vector index sits inside the same immutable Parquet superfile as the row data, the returned ids resolve against the same snapshot as scalar filters and BM25 results.

Why Sq16 sits between fp32 input and search

Infino accepts fp32 vectors. Internally, Sq16 represents each dimension with 16 bits. A 1024-dimension row drops from 4 KiB to about 2 KiB before surrounding metadata: roughly 40 GiB to 20 GiB across ten million vectors.

That reduction matters in both states. A resident HNSW walk touches fewer bytes per node. An OPANN query transfers and scores more candidates per object-store range. Full-precision input remains the API contract; quantization is an engine choice rather than a format the application has to manage.

Sq16 is still a quantized representation. Workloads requiring a different codec or direct control over every search parameter may prefer a dedicated vector database that exposes those choices. Infino chooses the representation and query parameters from the table’s data and serving state.

Serving state can change in place

The durable index remains in the Parquet superfiles. A hot working set can load a resident HNSW path over those vectors. A cold worker, a restart, or a table beyond the RAM budget continues through OPANN. Changing serving state does not create a second source of truth or require the application to reinsert vectors.

The object-storage path is the fallback when the resident graph is absent or evicted. Vector search becomes colder rather than unavailable because object storage still holds the data and the range-searchable index.

The measured object-storage path

On the 9.4-million-vector Cohere run (768 dimensions, top-10), the post-drain OPANN + Sq16 path reached 0.995 recall@10 with 9.37 ms p50 and 10.25 ms p99 warm latency. A cold open took a 661 ms median; the first cold query took 4.60 seconds while fetching 2.55 GiB of index ranges.

These measurements cover OPANN. They show the trade directly: planned range reads are fast once resident, but expensive on first touch. The resident HNSW path trades additional RAM and graph load/build cost for a query path that avoids the cluster scan.

Use the index that matches the medium

Different storage media favor different vector structures. HNSW turns resident memory into low-latency graph traversal. OPANN turns object storage into a small number of planned range reads. Sq16 lowers the bytes both paths have to move and score.

The shared invariant is the Parquet source beneath them: one durable vector corpus, with the serving path selected by where the working set lives.