DEV Community: Ashish Mishra

Choosing a database for AI agents: memory, retrieval and state compared

Ashish Mishra — Thu, 24 Sep 2026 00:00:00 +0000

Sep 24, 2026 · engineering

The right database for an AI agent depends on the jobs its memory has to do. An agent recalls past context by meaning, looks up exact names and identifiers, filters and counts structured state such as users, sessions and time windows, and keeps a history that only grows. Most agents need all of these at once, which is why keyword and vector search over the same records, with SQL over the results, has become the common requirement.

This page compares seven common options on those capabilities. It compares capabilities only, not prices, and it says where each option is the right choice.

What does an AI agent need from its database?

Recall by meaning. Vector search finds a past message or document that says the same thing in different words.
Recall by exact term. Names, ticket numbers, error codes and product IDs are where embeddings blur, because a rare token dissolves into its neighborhood. Keyword search with BM25 finds them. Where keyword search and vector search each fail shows both failures on real data.
Structured state. Memory is scoped to a user, a session, a tool or a time range, and an agent often needs a number rather than a list: how many times did this step fail this week, and for which customers.
Growth. An agent's history is appended to and rarely deleted, so the stored corpus grows every day while each query stays small.
Where it runs. A library inside the agent's process has no server to operate and no network hop. A server or hosted service can be shared by many agents and scaled on its own.

The first two are the case for hybrid search: an agent's queries mix exact identifiers with loosely worded recall, and each retriever misses what the other finds. The third is the case for SQL over the search results rather than a second system to join them in application code.

How the common options compare

system	runs as	keyword + vector in one query	SQL over results	storage format	open source
Infino	embedded library or hosted	yes: BM25 plus vector, fused with reciprocal rank fusion in one query	yes: DataFusion SQL, with search as table functions	Parquet, with the indexes inside the files	yes (Apache-2)
Postgres + pgvector	database server	partial: native full-text plus pgvector, BM25 needs another extension, ranks combined by you	yes, full Postgres SQL	Postgres tables	yes (PostgreSQL license)
Elasticsearch / OpenSearch	cluster or service	yes: BM25 plus kNN, fused	partial: ES\|QL, a SQL plugin, PPL	Lucene segments, engine-private	OpenSearch Apache-2, Elasticsearch AGPL / ELv2
Qdrant	server or cluster	partial: dense plus sparse vectors	no, retrieval API	engine-private indexes	yes (Apache-2)
Pinecone	hosted service	partial: dense plus sparse vectors	no, retrieval API	engine-private indexes	no, hosted only
LanceDB	embedded library or hosted	yes: full-text index plus vector, fused, optional reranker	partial: SQL filter expressions on queries	Lance, an open columnar format	yes (Apache-2)
Chroma	embedded, server, or hosted	partial: dense, sparse and full-text search, local and distributed features still converging	no, retrieval API with metadata filters	engine-private	yes (Apache-2)

Checked against each project's documentation in September 2026. Each project moves quickly, so check the current docs for the feature you depend on. The comparison pages go system by system.

When is each one the right choice?

Infino when an agent needs keyword, vector and SQL over the same memory in one query, with the data kept as Parquet in your own bucket and no server to run. It is not a system of record: transactional, write-heavy state belongs in an OLTP database next to it.
Postgres with pgvector when the agent's application state already lives in Postgres and the memory corpus is modest. One database, transactional writes and full SQL. The cost is that relevance fusion is yours to write, and text, vectors and the application share the same instances.
Elasticsearch or OpenSearch when full-text relevance tuning matters most, with analyzers, synonyms and mature ranking controls, and a team already runs a search cluster.
Qdrant or Pinecone when the workload is dedicated vector serving at a high sustained query rate, filtered on payload metadata. Pinecone when nobody should run servers at all.
LanceDB when the agent works with vectors and multimodal data on files, embedded in the process, and the same tables also feed training or data pipelines.
Chroma for the quickest local start on a prototype, with a hosted path when it grows.

What this looks like in Infino

Infino is an open source retrieval library that runs inside the agent's process. A superfile is a search index that is a valid Parquet file: the data and the BM25 and vector indexes in one file. An agent's memory is one table of those files, on local disk or in object storage, and a single query can run a hybrid search, filter it by user and time, and group the results, because each search function is a table in SQL.

The agent memory guide walks through the pattern with code, and agents that speak MCP can use the Infino MCP server without writing any. Implementing hybrid search on Apache Parquet files is the runnable end-to-end version.

Common questions

Do AI agents need a vector database?

They need vector search, which is not the same thing. Vector search recalls past context by meaning, but it misses exact names, identifiers and error codes, and a standalone vector database adds a second system to keep in sync with the rest of the agent’s state. Engines that run keyword and vector search over the same records cover both needs.

Is hybrid search necessary for agent memory?

For most agents, yes. Their queries mix exact identifiers, such as a customer name or a ticket number, with loosely worded recall, such as what went wrong last time. Keyword search finds the first kind and vector search finds the second, and fusing the two rankings returns the rows that either one alone would miss. How hybrid search fuses the two.

Can I use Postgres for AI agent memory?

Yes. Postgres with pgvector and its native full-text search covers agent memory well when the application already runs on Postgres and the corpus is modest. BM25 ranking needs another extension, the two rankings are combined in your own SQL, and memory shares the instances that serve the application.

Should an agent’s database be embedded or a server?

Embedded when one agent process owns its memory and simplicity matters, since there is nothing to deploy and no network hop on a lookup. A server or hosted service when many agents share one store or it needs to scale apart from them. Some engines, Infino and LanceDB among them, offer both.

What storage format should agent memory use?

An open one, if other tools need to read the history. Engine-private formats are readable only through the engine that wrote them. Infino stores memory as standard Parquet, so any Parquet reader can open it for analytics or export without Infino in the path. Parquet interop.

Open VDBBench: a fair and open benchmark for vector databases

Ashish Mishra — Thu, 24 Sep 2026 00:00:00 +0000

Sep 24, 2026 · engineering

If you want to know which vector database is fastest, you probably end up at VectorDBBench. Most engineers check it, and it’s the benchmark AI assistants cite when you ask them. Zilliz, the company behind Milvus and Zilliz Cloud, maintains the harness and runs the leaderboard at zilliz.com/vdbbench-leaderboard. Zilliz products hold the top three places on it.

When we submitted a client for Infino last month, we went back through the repo history to see how results make it onto that board, and found serious integrity problems. Most of the merged clients have never had a result published. Zilliz’s own products get re-run on each new release, while competitors sit on versions more than a year old, some with their compression switched off. And the dates and versions that would show you this were removed from the leaderboard data. Read the full critique for the PRs and configs behind each point.

We wanted a board where every engine runs its current version with compression on, and every number says when it was measured. So we forked the harness and ran the engines ourselves. The results are live at infino.ai/open-vdbbench.

How Open VDBBench runs

Open VDBBench uses the same VectorDBBench cases, Cohere 1M and Cohere 10M at 768 dimensions, with our fork of the harness. Each engine gets its own Azure Standard_D16ads_v7 (16 vCPU, 64 GB, local NVMe), with the client on the same VM. We sweep each engine’s search parameter and rank it by its highest QPS at recall ≥ 0.90. If you care about higher recall, the board can re-pick every engine at 0.95 or 0.99.

Every row carries the date it was measured and the version the runner read from the engine itself. If an engine’s client supports a quantized index, we use it. Anything that runs gets published, whatever it scores, and every row links to the result file it came from. The full rules are in METHODOLOGY.md.

Here’s how three engines ran on each board:

	zilliz board	open board
Elasticsearch	8.17, HNSW float32, rescore off	9.5.1, int8_hnsw, rescore, oversample 2.0
OpenSearch	2.17, `use_quant: false`	3.8.0, HNSW with fp16 scalar quantization
Milvus	2.6.14, SQ4U+FP16 and SQ8	2.6.14, HNSW_SQ, SQ4U with FP16 refinement

The first pass covers 19 engines at 1M and 13 at 10M. Infino runs under the same rules as everyone else, and it isn’t first at 1M. OceanBase is.

Add your database

If your database is missing or on an old version, send a pull request to infino-ai/open-vdbbench. It needs an entry in harness/matrix.json pinned to an exact image tag, plus bring-up scripts if the database is new. A version bump only changes the tag.

We’ll merge every new database within a month of its pull request. Every month, we run each new and updated database at 1M and 10M on the reference machine and put the results on the board.

You don’t need two databases for hybrid search

Ashish Mishra — Wed, 23 Sep 2026 00:00:00 +0000

Sep 23, 2026 · engineering

Keyword search and vector search can run in one engine over one copy of your data. Infino is an open source retrieval library that keeps the text, the embeddings, and both the BM25 and vector indexes in one table of Parquet files on object storage, and a single hybrid query returns one ranked result set. There is no second database to feed and no sync job between them.

Most teams arrive at the same setup by stages.

They start with embeddings and a vector database because semantic search feels like the new part. Then the misses show up. Error codes disappear. Part numbers blur together. Proper names drift. So they add keyword search beside it, and now retrieval works well enough.

It also means they are running two search systems for one job.

That part gets waved away too easily. People spend their time on fusion weights and ranking tricks because that work is visible. The expensive part sits underneath it. Two services to run. Two copies of the corpus. Two indexing paths. One sync loop that nobody wants to own and everybody depends on.

Hybrid search is not a luxury feature. It is what you end up building once you learn where each method breaks.

Embeddings are good at meaning. They are weak at exact strings. BM25 is good at exact terms and bad at semantic drift. If your corpus contains manuals, tickets, logs, or internal docs, you usually need both. We ran both retrievers over 10,003 real banking questions in where keyword search and vector search each fail, and what is hybrid search covers the fusion math.

The older assumption was that this meant two different systems. That assumption came from the period when vector search had to be bolted onto the stack from the outside. It no longer needs to be.

What gets expensive in the usual stack

Look at the operational shape of the common design.

You keep one system for vectors and another for full-text. You write every document into both. You keep two schemas in step. You watch two dashboards. You debug two failure modes. When one side lags, retrieval quality drops in ways that are hard to spot and annoying to explain.

The cost model is just as awkward. Both systems charge you for stored corpus size. The sync path costs you on every change. Query traffic matters less than people expect. Double the corpus and you often double the retrieval bill, even if traffic stays flat.

The ranking code is not the real burden here. Most teams can fuse two result lists in an afternoon. The part that lingers is everything around it.

Open source hybrid search on object storage

Infino takes the obvious route and keeps one copy. It is an open source retrieval library under Apache-2.0, embedded in your process rather than run as a cluster.

The table holds the text, the embeddings, and the indexes for both BM25 and vector search. A hybrid query runs over the same rows and returns one ranked result set. There is no second database to feed and no merge step in the application. That matters less because it saves a few milliseconds and more because it removes an entire layer of operational drag.

There is another problem teams feel later. Once the index format is proprietary, the data becomes sticky. Leaving means re-indexing everything, moving the corpus, and often rebuilding the serving path around it. The migration cost grows with time.

Infino cuts against that from both directions. The engine is open source, so nothing in the read path is a black box, and the storage format is one you already use. A superfile is a search index that is a valid Parquet file: the data and the BM25 and vector indexes in one file (core concepts). Those files sit in your own bucket, and any Parquet reader opens one as an ordinary table.

Storage is priced like object storage, not like hot memory reserved for the life of the corpus.

Readers do not need to hold the full dataset. They scale with query load rather than raw corpus size. A top-k lookup turns into a small set of offsets and range reads, with local caching where it helps.

This is the right shape for a lot of RAG systems. The corpus grows steadily. Queries are small. Most documents sit cold most of the time.

What it looks like in code

retrieval.py

import infino
import pyarrow as pa

db = infino.connect("s3://my-bucket/docs")

schema = pa.schema([
    pa.field("body", pa.large_utf8(), nullable=False),
    pa.field("embedding", pa.list_(pa.float32(), 384), nullable=False),
])

docs = db.create_table(
    "docs", schema,
    infino.IndexSpec().fts("body").vector("embedding", 384, "cosine"),
)

docs.append(rows)

hits = docs.hybrid_search(
    "body", "E-4021 after rotating a key",
    "embedding", query_vec,
    k=10,
    projection=["_id", "body", "score"],
)

The text and the embeddings live on the same rows. BM25 and vector search run in one place. The caller gets back one ranked set of Arrow rows, with projection choosing the columns (it returns _id and score alone if you leave it off). The search guide has every retrieval mode with runnable code, including how RRF fuses the two rankings.

The environment switch is small:

connect.py

infino.connect("memory://") # tests
infino.connect("./data") # laptop, no server
infino.connect("s3://my-bucket/docs") # your bucket, queried in place

Infino does not generate embeddings for you. Bring your own model, write the vectors you want, and query with the vectors you already trust.

If you use a reranker, keep using it. Retrieval and reranking are different jobs. Pull back a wider candidate set, rerank it with a cross-encoder, and pass the short list to the model. Nothing about a single-engine hybrid design changes that.

This is not a claim that every search workload belongs on object storage.

If you need an in-memory serving tier with every knob exposed and corpus size is not the problem, there are good tools for that. If your pain comes from running two systems for one retrieval path, the trade-off looks different. Then the question is less about peak benchmark numbers and more about how much machinery you are carrying just to issue a search.

That is the case Infino is built for.

One table. One copy of the corpus. Hybrid retrieval over your own files. SQL in the same process when you need it.

You can try it locally with a folder and move the same data to object storage later.

install.sh

pip install infino

The engine is on GitHub at infino-ai/infino.

Common questions

Do I need a separate vector database for hybrid search?

No. The vector index and the BM25 index can live in the same table as the text. Infino writes both into the same Parquet files and hybrid_search queries them together, so there is no second system to feed or keep in sync. The search guide.

What does running a vector database next to a search cluster cost?

Two copies of the corpus, two indexing paths, two schemas kept in step, and a sync job between them. Both systems usually charge by stored corpus size, so doubling the corpus tends to double the retrieval bill even when query traffic stays flat.

Where does the data live?

In your own bucket as Parquet files, or on local disk. The same code runs against memory:// for tests, a local folder, or an s3:// prefix, and any Parquet reader can open the files as an ordinary table. Parquet interop.

Does Infino generate embeddings?

No. Bring your own model, write the vectors with the rows, and query with a vector from the same model.

Is Infino open source?

Yes. The engine is Apache-2.0 licensed and runs as a library inside your process rather than as a cluster. infino-ai/infino on GitHub.

Where keyword search and vector search each fail

Ashish Mishra — Tue, 08 Sep 2026 00:00:00 +0000

Sep 8, 2026 · engineering

We indexed 10,003 real questions that customers asked an online bank and ran the same queries through BM25 and vector search. The failures are instructive in both directions, and they are the whole case for hybrid search. This post shows the failures. The full walkthrough in the docs is runnable end to end: dataset, embeddings, index build, every query shape, and a read-back of the produced files with a plain Parquet reader.

The paraphrase that breaks keyword search

BM25 ranks rows by the exact tokens they share with the query. A user asking about a card that never arrived phrases it their own way, and BM25 anchors on "not shown up" and drifts into refund territory by the second hit:

keyword_miss.py

table.bm25_search("text", "my new card still has not shown up", 5)

17.39 Why has my new card still not come? 15.52 My refund has not shown up on my statement. 15.15 my check or cash deposit has not shown up 14.04 My refund has not shown up in my account. Where is it? 13.42 My statement has not shown my refund.

Vector search matches meaning, so the same paraphrase works even where a hit shares almost no vocabulary with the query:

semantic.py

table.vector_search("embedding", qvec, 5, projection=["text", "score"])

0.117 Why has my new card still not come? 0.165 why have i not got my new card? 0.172 I just got a new card how do I get it to start working? 0.180 where is my new card? 0.181 2 weeks ago I ordered my new card. It isn't here. What should I do?

The token that breaks vector search

Invert the query and the failure inverts with it. On a term of art, BM25 is precise:

keyword.py

table.bm25_search("text", "SWIFT transfer", 5, projection=["text", "score"])

9.46 Is a SWIFT transfer acceptable? 9.46 Can I transfer using SWIFT? 9.46 Can I Transfer from SWIFT? 9.08 Can I do a SWIFT transfer? 8.72 is there a fee for swift transfer

Vector search has no notion of an exact token. A rare identifier, a product name, or an error code can dissolve into its embedding neighborhood, outranked by rows that are merely on topic. A real query stream contains both shapes daily. Whichever single retriever you pick, some of your users lose.

Run both, fuse the rankings

Hybrid search runs both retrievers and merges the two ranked lists with reciprocal rank fusion, so a row both retrievers surface outranks a row only one of them found:

hybrid.py

q = "top up with Apple Pay"
table.hybrid_search("text", q, "embedding", model.encode(q).tolist(), 5,
                    projection=["text", "score"])

0.0323 Is top-up possible with Apple Pay? 0.0320 How can I top up with Apple Pay? 0.0308 Am I able to top up with Apple Pay? 0.0161 My Apple Pay top up with my American Express card is not working? 0.0161 is there a way to do top up with apple pay

The exact-token half of the query (Apple Pay) and the intent half (topping up) each pull their weight. Neither retriever alone ranks this list this way. If you want the concept in depth, what is hybrid search? covers the two retrievers and the fusion math.

All of this ran directly on Apache Parquet files. The index lives inside the files, they stay readable by any plain Parquet reader with Infino nowhere in the read path, and the same code targets S3 by changing one string. The docs guide is the complete walkthrough: install, dataset, embeddings, the index build, filtered search, SQL composition over search results, and the read-back. Inside a Parquet superfile covers how the format carries an index without breaking compatibility.

Fixed grids on vector search - comparing 3 approaches to quantization: FAISS, Turbovec, and Infino

Ashish Mishra — Tue, 01 Sep 2026 00:00:00 +0000

Sep 1, 2026 · engineering

We benchmarked three approaches to 4-bit quantized vector search over the same 100,000 OpenAI embeddings (1536 dimensions): FAISS's classic product quantization, turbovec — an open-source implementation of TurboQuant — and Infino's SQ4, which we build, each measured through its own public API against exact brute-force ground truth. At 4 bits per dimension every index stores 768 bytes of codes per vector, and all three sit between 0.94 and 0.97 recall — but the latencies run from 1.5 ms to 45 ms.

The sections below work through where that difference comes from, build and write cost, and the same scan at four corpus sizes.

Quantization, briefly

There are two ways to make approximate search cheaper, and they're independent of each other.

a. You can look at fewer vectors. That's routing : IVF (inverted file) groups the corpus into clusters and reads only the clusters nearest your query, HNSW (hierarchical navigable small world) walks a graph toward the answer, tree methods carve up the space.

b. Or you can make each vector cheaper to look at. That's quantization , and it's the only thing that varies here — every index in this comparison scans all 100,000 vectors on every query. The FAISS index string carries no coarse quantizer, the turbovec index is a flat scan by design, and the Infino mode we measured is an exhaustive scan over its in-memory codes.

Scalar quantization treats each coordinate on its own: take the float, round it to one of 16 levels, store 4 bits. 1536 coordinates, 768 bytes. Usually there's a rotation first, and those 16 levels can be evenly spaced or fitted to the data — Lloyd-Max, the classic minimum-error placement, is one such fit. Product quantization chops the vector into M pieces and runs k-means separately on each piece, replacing that chunk of the vector with the index of its nearest centroid. The list of centroids a code indexes into is called a codebook, and each piece's codebook is a subquantizer. Because the centroids live in the chunk's own space, they can capture correlation between coordinates in a chunk, which per-coordinate rounding cannot. The price is a training pass over the corpus.

One thing none of these indexes does: rerank. In many systems you take the top few hundred results then re-score those against the uncompressed vectors.

The three approaches

FAISS is Meta's vector library, and its indexes are specified by factory string. The one here is IDMap,PQ768x8np — classic product quantization with a lookup table: 768 subquantizers of two dimensions each, so one 8-bit code covers two coordinates and the budget is the same 4 bits per dimension and 768 bytes per vector as everything else in the article. It is also the configuration turbovec's own README benchmarks its recall against.

turbovec implements TurboQuant, and we ran it at its 1.0.0 release. No routing structure at all, a flat scan over compressed vectors. Per its docs, each vector is rotated, quantized to 2 or 4 bits per coordinate against a Lloyd-Max codebook, and renormalized by length, and the search kernels use nibble-split lookup tables over an interleaved layout, chosen at runtime.

The TurboQuant authors' own released code ships only a conceptual Python path for inner-product estimation — no optimized scan, which is why the RaBitQ team's comparison report excluded query-time efficiency entirely. As far as we know turbovec is the only way to measure this method's scan at full speed.

Infino is an embedded retrieval library whose tables are Parquet files with the vector index embedded in them. It doesn't implement one fixed vector algorithm. The index serves through one of several modes — a routed cluster scan by default, an in-memory graph, or a flat scan — picked by a config setting, and optimize() sizes whichever is selected against a recall target measured on the table's own data. The mode tested here is the flat scan, flat_ivf: a 4-bit plane scanned exhaustively. At small corpora it's the effective shape and it's the configuration comparable to a flat library scan.

The setup

Everything runs in one process over 100,000 vectors from the dbpedia OpenAI corpus — 1536 dimensions, cosine — each engine through its own public API. The machine is an EPYC 9V74 slice: 4 cores, 8 hardware threads, 63 GiB of RAM.

Resident memory at 100,000 vectors: fp32 against the 4-bit index.

At fp32, this corpus is 586 MiB of vectors. Every index below serves it from about 75 MiB — 4 bits per dimension in place of 32. Quantization writeups often make this the headline — turbovec's README included, where the compression figures ("31 GB of RAM as float32… fits it in 4 GB", "16x compression") are quoted against full 32-bit floating point vectors. The reduction in memory is real, but it is also what any 2- or 4-bit quantization delivers.

To measure recall, ground truth is exact brute-force nearest neighbors, computed once and reused. Recall is graded at k = 1, 10 and 100 over 1,000 held-out queries, and every engine answers the same query set.

Latency, memory and recall

Queries run one at a time, so the latency measurements are per-query scan costs and say nothing about throughput. Nothing pins threads: each engine parallelizes a single query however it chooses.

	recall@10	p50	bytes/vector	resident
FAISS flat PQ	0.950	45.5 ms	791	75.5 MiB
turbovec 4-bit	0.948	1.59 ms	788	75.2 MiB
Infino 4-bit	0.963	1.50 ms	788	75.2 MiB

Infino is slightly faster than turbovec, and both are significantly faster than FAISS.

As expected, memory differences are a wash. 1536 dimensions at 4 bits is 768 bytes of codes, and all three indexes land within 3% of that, at 788 to 791 bytes per vector.

The rest is bookkeeping. turbovec and Infino each store 20 bytes: an id plus a per-vector scalar that corrects the quantized dot's bias. Flat PQ stores an 8-byte id plus its trained codebook, which is 1.5 MiB shared across 100,000 vectors, or 15.7 bytes each — 768 + 8 + 15.7 = 791.7, against 791 measured.

Recall at k = 1, 10, 100:

	k=1	k=10	k=100
FAISS flat PQ	0.949	0.950	0.963
turbovec 4-bit	0.946	0.948	0.957
Infino 4-bit	0.970	0.963	0.970

Every number here is graded over 1,000 held-out queries, which puts the standard error on a mean recall near ±0.004. FAISS PQ and turbovec are within a couple of points of each other at every k. Infino leads the group by at least 1.3 to 1.8 points at every size we ran.

Searching the vector space

A 4-bit code can take sixteen values, so every quantizer here has sixteen levels to place. The three approaches place them differently.

What each approach trains, stores, and scores with.

FAISS trains centroids. It runs k-means in 768 two-dimensional subspaces, 256 centroids each — that's the 312 CPU-seconds of build time, and it buys a codebook that can represent correlation between paired coordinates, which neither one-dimensional fit in this comparison can. The scan never compares the query against the compressed vectors directly: once per query it precomputes the distance to every centroid — a 768 KiB table, 768 × 256 floats — then scores a candidate by summing one lookup per subquantizer, 768 of them, each at an address not known until the code byte has been read.

Turbovec precomputes its levels. The insight behind it is not TurboQuant's alone — rotate-then-quantize is a family: RaBitQ (SIGMOD 2024) established the construction and its error bounds although there is an ongoing discussion on precedence. The property both rely on is the same: after a random rotation, every coordinate follows the same known distribution — a Beta, near-Gaussian in high dimension — so the optimal 16-level Lloyd-Max quantizer for that distribution can be computed once, ahead of time, and reused for every corpus forever. (Infino's ruler, next, is the same construction with uniform levels.) Nothing is trained and nothing is fitted; the 1.2 CPU-seconds is rotating and encoding, which is the point — the paper calls it data-oblivious: no pass over the data is ever needed, so a vector can be quantized the moment it arrives. turbovec also stores one scalar per vector that corrects the inner product's quantization bias. At scan time the fitted levels are read through lookup tables of 16 entries per coordinate — small enough to stay in SIMD registers — at 1.59 ms.

Infino computes a ruler based on the data. It rotates (a seeded structured rotation — only the 8-byte seed is persisted), then fits offset and step per rotated coordinate over mean ± 2.7σ , uniform steps between. No codebook at all. The 2.7 is the optimal 16-level loading for a Gaussian, and the rotation is what makes the Gaussian assumption hold: it spreads each coordinate toward the same near-Gaussian marginal, which is what lets one global ruler — a fixed grid, the same sixteen levels for every row in the table — work at all; on the raw embedding axes, with their wildly uneven energy, most coordinates would collapse onto a couple of codes. A comment in the encoder records the measured stakes: loading over sigma instead of min/max moved recall@10 up 0.032 on this corpus, because a min/max ruler is set by each coordinate's single most extreme value and spends most of its 16 levels on range almost no row occupies. The scan carries no distance table of any size — because the levels are evenly spaced, scoring reduces to a direct integer dot on the packed nibbles — and runs at 1.50 ms.

Building the vectors

	wall (8 threads)	CPU-seconds
turbovec 4-bit	374 ms	1.2
Infino¹	1.7 s	—
FAISS flat PQ	39.7 s	312

¹ Infino's build is a table write: the vectors are appended and committed, so the 1.7 s ends with the data durable on disk as Parquet, where the library builds end in RAM. With a single writer it's 3.57 s. The harness doesn't record CPU-seconds for it.

FAISS trains: 768 k-means problems, 256 centroids each over a two-dimensional subspace, iterated over a sample of the corpus — that's the 312 CPU-seconds. TurboQuant's codebook is precomputed for the rotated coordinate distribution and never reads the corpus; turbovec's 1.2 CPU-seconds is rotation and encoding. Infino's build trains too — the 1.7 s includes its own k-means — but we didn't profile it by stage.

On this corpus the trained codebook and the precomputed one reach the same recall at the same bytes. That's corpus-dependent, and we ran only the one. We didn't test anything with strong subspace structure, which is exactly where trained codebooks would have more to fit than a per-coordinate scheme does.

A new vector has two jobs: get into the search structure, and get onto disk. FAISS and turbovec split them. Their add writes straight into the layout the scan kernels read, so after turbovec's 18 µs the vector is searchable — in RAM, where a crash takes it back, because nothing touches disk until save rewrites the whole index. Infino's append does both jobs in one call: when it returns, the rows are searchable and committed.

	add	remove	persist	full cycle
FAISS flat PQ	3.14 ms	401 µs	15.2 ms	68.5 ms
turbovec 4-bit	18.0 µs	3.4 µs	33.4 ms	21.4 ms
Infino 4-bit	27 µs¹	2.7 µs¹	—	17.0 ms

¹ per row, 100,000 rows sharing one commit, for the append and the delete alike. Infino separates add and save operations internally but renders them combined in its public API: append and delete each do the in-memory work and the save in one call, so the per-row cost is set by how many rows share it. A row alone in the commit costs the full 17 ms to add, 7.9 ms to delete.

Deletes have the same latency shape — turbovec's 3 µs remove is the in-RAM half, durable on its next save, and Infino's is again a commit — with one twist: a tombstone — the committed delete marker — carries no vector payload, so a wide delete amortizes below the in-RAM number, 2.7 µs a row durable against 3.4 µs in RAM. The two write models come down to what gets batched: the libraries batch time, with mutations waiting in RAM until the next save and a crash losing the interval, while Infino batches rows, and every call is a durability boundary.

Committing every write costs the scan nothing: 788 bytes per vector and 1.50 ms, inside the range of the two libraries.

Scaling the quantization

Everything above shares one constraint: to answer a query, each of these indexes reads every row. A SIMD scan over contiguous bytes has the DRAM and L1/L2/L3 cache behavior that random pointers through data structures can't match, whether it is a graph, a tree, or other types of routing. At small corpora, flat scan is the natural shape.

We ran the same benchmark at four corpus sizes:

warm p50 @ k=10	100K	250K	500K	933K
Infino SQ4	1.50 ms	4.64 ms	9.90 ms	18.88 ms
turbovec 4-bit	1.59 ms	4.84 ms	10.19 ms	19.05 ms
FAISS flat PQ	45.5 ms	111.8 ms	226.3 ms	414.7 ms

Recall over the same sweep:

recall@10	100K	250K	500K	933K
Infino SQ4	0.963	0.962	0.960	0.960
turbovec 4-bit	0.948	0.945	0.941	0.937
FAISS flat PQ	0.950	0.945	0.942	0.942

Infino recall holds within 0.3 points across the sweep. turbovec and flat PQ drift down about a point. However, given the linear nature of scans all 3 libraries incur latency penalties as the corpus scales. In the next post we'll show how we scale past that point, with different ANN (approximate nearest neighbor) structures built on the same fixed-grid philosophy.

Limits

FAISS also ships a SIMD variant of this index: PQ1536x4fs, "FastScan," which trades the float lookup tables for 8-bit-quantized ones resolved by shuffle instructions. We measured it, and its speed is real — 4.4 ms on this corpus, 10× faster than the classic scanner. Its recall is not publishable: at this 1536-subquantizer geometry it swung between 0.59 and 0.91 across corpus sizes in our runs, and a control that scored the identical trained codebook through the classic float tables read a steady 0.934 — so the loss is inside the fast-scan kernel, not the quantization. We found no published comparison that reports FastScan recall at all; turbovec's own README uses classic PQ as its recall baseline and cites FastScan only as a speed reference, and FAISS normally deploys it with a reranking stage.
FAISS PQ is normally deployed at 32 to 96 bytes per vector, where its lookup table would be 32 to 96 KiB; the 768 bytes per vector measured here is far outside its usual range.
Queries run one at a time throughout, so nothing in the test measures concurrency.
Every recall number is a raw quantized operating point; we didn't measure any of these indexes with a rerank stage.

Reproducing it

reproduce.sh

git clone https://github.com/infino-ai/retrievalbench && cd retrievalbench
printf 'vector:\n search_mode: flat_ivf\n' > infino.yaml
INFINO_BENCH_SUPERTABLE_DOCS=100000 \
  cargo bench -- vector-codec \
  corpus=hf:KShivendu/dbpedia-entities-openai-1M corpus-dir=./corpora

The scaling table is the same command at INFINO_BENCH_SUPERTABLE_DOCS of 250000, 500000, and 1000000; the recall and latency tables all come from that one four-size sweep, and the build and write tables from the same harness at 100K. The corpus downloads once from HuggingFace. The FAISS row needs --features faiss after scripts/build_faiss.sh, which builds the bundled FAISS source with -march=native — without it, FAISS's SIMD kernels silently fall back to scalar code. The engine dependencies are pinned to a commit, and every committed run records host, engine commit and command in a run.json beside the numbers under results/inprocess/.

What is hybrid search?

Ashish Mishra — Tue, 25 Aug 2026 00:00:00 +0000

Aug 25, 2026 · engineering

Hybrid search runs two retrievers over one query. A keyword retriever scores documents with BM25 against an inverted index. A vector retriever finds nearest neighbors to an embedding of the query. Each returns its own ranked list, and a fusion step merges the two into one ranking.

The reason to run both is that they miss different things.

Two retrievers, opposite failure modes

BM25 is literal. It finds the error code, the part number, the surname, the exact phrase someone typed, and it misses the same idea written another way.

Vector search is about meaning. It finds the paraphrase, the synonym, the other language, and it will also hand back things that are merely on topic.

A query like retry a failed request needs both: the token retry, and the idea of backoff, try-again, transient failure.

A question in clauses

Take “Which services retried a failed request, by team?”. Each clause asks for something different.

Which services is a count: COUNT(DISTINCT service).
By team is a GROUP BY: one count per team.
Retried is a word to match. BM25 finds the events that say it.
A failed request has no fixed wording. Vector search finds “try again”, “transient failure”, “exponential backoff”.

Fusing two incompatible scores

BM25 similarity rises for a better match, while vector distance falls. The two numbers are not on the same scale and adding them means inventing a weight. Reciprocal rank fusion sidesteps that by using position in each list instead of the scores themselves.

rrf.txt

contribution(rank) = 1 / (60 + rank)

# first in a list contributes 1/61
# a document first on both sides: 1/61 + 1/61 ≈ 0.0328
# first on one side only: 1/61 ≈ 0.0164
# tenth on both sides: 1/70 + 1/70 ≈ 0.0286

Each list is best-first. A hit at rank r (first place is 1) contributes 1 / (60 + r). The two contributions add, and the sum is the emitted score. A document both retrievers liked climbs above a document that only one of them ranked first.

60 is the constant from the 2009 paper that introduced the method, where fusing runs this way beat every individual system and Condorcet Fuse. It keeps fusion deterministic across queries.

Fusion needs a stable identity to merge on. A row's _id is what makes the same document one document, even when the two indexes store it differently.

k controls recall

Each lane retrieves k hits and fusion keeps the top k by fused score. A result LIMIT is a separate number: how many of those rows the query returns.

A result page uses a modest k (50-200) and a small LIMIT.
An aggregate over matching events uses a large k (thousands) so GROUP BY and COUNT see the full candidate cohort.
The disagreement set (rows only one retriever found) also needs a deep k. Shallow k hides the misses.
Raising k costs work on both lanes and increases retrieval depth without changing fusion weights.

Seeing where the two disagree

Fusion hides the split by design. To measure it, run each retriever on its own and join the two relations on _id. The rows on only one side are the ones a single retriever never returns.

disagreement.sql

-- where do keyword and meaning disagree, and how much does each add

WITH sides AS (
  SELECT CASE
           WHEN v._id IS NULL THEN 'keyword only' -- literal, no paraphrase
           WHEN k._id IS NULL THEN 'meaning only' -- the same thing, said differently
           ELSE 'both agreed'
         END AS found_by,
         k.score AS bm25,
         v.score AS cosine
  FROM bm25_search('tickets', 'body', 'disk full on ingest', 100) k
  FULL OUTER JOIN
         vector_search('tickets', 'embedding', :q, 100) v
      ON v._id = k._id -- one snapshot, both sides
)
SELECT found_by,
         count(*) AS docs,
         round(avg(bm25), 3) AS avg_bm25, -- nulls skip themselves
         round(avg(cosine), 3) AS avg_cosine
FROM sides
GROUP BY found_by
ORDER BY docs DESC;

Run it per query class. When the “meaning only” bucket grows, the embeddings and the vocabulary are drifting apart. Note that k.score is BM25 (higher is better) while v.score is a distance (lower is better), which is the reason fusion compares rank positions rather than adding those columns.

What sits around retrieval

Hybrid search is the retrieval middle: one keyword query, one vector query, fusion, a ranked list. The quality stages on either side of it belong to the application.

Query rewrite sits in front. Several rewritten queries can run as separate searches or a UNION.
A cross-encoder reranker sits after, on the dozens of rows that came back. Rescoring candidates inside an ANN index is a different thing that shares the name.
Chunking, contextual retrieval, and a labeled eval set are ingest and evaluation work.

In Infino, both indexes live inside the same Parquet files, both lanes read one pinned snapshot, and the fused result is a SQL relation: hybrid search on Infino · the query surface.

Why object-storage vectors use OPANN + Sq16, while RAM uses HNSW

Ashish Mishra — Thu, 20 Aug 2026 00:00:00 +0000

Aug 20, 2026 · engineering

HNSW is an excellent in-memory vector index. It is also a poor object-storage access pattern. Each graph hop depends on the previous hop, so a cold walk turns pointer chasing into serialized range requests.

Infino therefore uses two serving paths over the same vector data: HNSW when the working set is resident in RAM, and OPANN (Infino’s object-storage ANN path) with Sq16 when the vectors are served from Parquet on object storage.

The same corpus has two hardware states

A 1024-dimension fp32 vector occupies 4 KiB before index metadata. Ten million vectors are roughly 40 GiB of raw values. If that working set and its graph fit in the available RAM, spending memory to minimize query CPU and latency is reasonable.

The same decision fails for a large or mostly cold corpus. Keeping every vector and graph edge resident makes storage cost scale with the full dataset, even when only a small fraction receives queries. Moving the graph itself to object storage avoids the RAM bill but replaces memory loads with dependent network reads.

two-serving-states.txt

hot working set
  query ─▶ HNSW graph walk ─▶ top-k
          random access in RAM

cold / object-storage working set
  query ─▶ OPANN routing ─▶ bounded cluster ranges ─▶ top-k
                           contiguous object reads

durable source for both
  parquet superfiles in object storage

HNSW: random access is a feature in RAM

HNSW builds a navigable graph over the vectors. A query starts at an upper layer, follows promising neighbors toward the query, and expands a candidate set near the bottom. The graph avoids scoring most vectors.

In RAM, those dependent hops are cheap. The next node address may be unpredictable, but it resolves as a memory access rather than a storage request. A graph over the resident Sq16 representation also reduces the bytes touched per score relative to fp32.

The cost moves to build time and memory. The graph stores edges in addition to vector data, and loading or rebuilding it is work a cold worker must pay before it receives the low-latency path. HNSW is the right answer only when the working set is actually resident.

Why graph hops do not belong on object storage

Object storage rewards planned reads. One request can fetch a useful contiguous range at high throughput, but every additional dependent request pays request latency again.

An HNSW walk cannot issue all of its reads in advance: the node selected at step N determines which neighbors exist at step N+1. Even with caching, a cold graph walk creates the exact sequence object storage handles poorly: small, unpredictable, serialized reads.

Downloading the whole graph before searching avoids that request pattern, but turns the first query into a bulk load and requires enough local capacity for every cold namespace. That is pinning under another name.

OPANN: route first, then read contiguous ranges

The object-storage path groups vectors into regions that can be addressed independently. A small routing structure chooses the regions most likely to contain the nearest results. The query then fetches those cluster ranges, scores their compact vectors, and reranks a shortlist.

The important property is that the expensive reads are known before the data arrives. Cluster ranges can be requested concurrently, coalesced, cached, and reused. A query touches a bounded subset of the vector region instead of traversing a graph one network-dependent node at a time.

Because the vector index sits inside the same immutable Parquet superfile as the row data, the returned ids resolve against the same snapshot as scalar filters and BM25 results.

Why Sq16 sits between fp32 input and search

Infino accepts fp32 vectors. Internally, Sq16 represents each dimension with 16 bits. A 1024-dimension row drops from 4 KiB to about 2 KiB before surrounding metadata: roughly 40 GiB to 20 GiB across ten million vectors.

That reduction matters in both states. A resident HNSW walk touches fewer bytes per node. An OPANN query transfers and scores more candidates per object-store range. Full-precision input remains the API contract; quantization is an engine choice rather than a format the application has to manage.

Sq16 is still a quantized representation. Workloads requiring a different codec or direct control over every search parameter may prefer a dedicated vector database that exposes those choices. Infino chooses the representation and query parameters from the table’s data and serving state.

Serving state can change in place

The durable index remains in the Parquet superfiles. A hot working set can load a resident HNSW path over those vectors. A cold worker, a restart, or a table beyond the RAM budget continues through OPANN. Changing serving state does not create a second source of truth or require the application to reinsert vectors.

The object-storage path is the fallback when the resident graph is absent or evicted. Vector search becomes colder rather than unavailable because object storage still holds the data and the range-searchable index.

The measured object-storage path

On the 9.4-million-vector Cohere run (768 dimensions, top-10), the post-drain OPANN + Sq16 path reached 0.995 recall@10 with 9.37 ms p50 and 10.25 ms p99 warm latency. A cold open took a 661 ms median; the first cold query took 4.60 seconds while fetching 2.55 GiB of index ranges.

These measurements cover OPANN. They show the trade directly: planned range reads are fast once resident, but expensive on first touch. The resident HNSW path trades additional RAM and graph load/build cost for a query path that avoids the cluster scan.

Use the index that matches the medium

Different storage media favor different vector structures. HNSW turns resident memory into low-latency graph traversal. OPANN turns object storage into a small number of planned range reads. Sq16 lowers the bytes both paths have to move and score.

The shared invariant is the Parquet source beneath them: one durable vector corpus, with the serving path selected by where the working set lives.

Infino's self-driving vector engine

Ashish Mishra — Fri, 07 Aug 2026 00:00:00 +0000

Aug 7, 2026 · engineering

You write a table. Infino builds the vector index — and tunes it to your data. No n_cent, no nprobe, no rerank_mult, no sweeps. On Cohere-10M, that's 0.99 recall@10 — with zero knobs touched.

Write a table; Infino inverts it in the background into a distance-aligned index, then stamps the ANN dials — n_cent, nprobe, rerank — from your own data. No sweeps.

Most vector databases make you hand-tune dials such as n_cent, nprobe, ef_search, or rerank depth — and wish you luck. Get them wrong and you burn compute or silently ship poor recall, and the "right" values shift with scale and data.

The dials that run your recall bill

n_cent — how finely to partition the index
nprobe — how many partitions to actually search
rerank multiplier — how many candidates to re-score exactly

Set them too conservative and you burn compute chasing recall you didn't need. Too aggressive and you silently ship 0.6 recall to users who expected 0.99. And there's no universal right answer: the sweet spot moves with corpus size, embedding distribution, even your query mix. Perfect at 1M is wrong at 100M. Perfect for clean product vectors, broken for messy multilingual text.

So teams run sweeps, hard-code a compromise, and re-tune every time the data grows. Tuning your ANN index becomes a standing job — a tax on every vector workload.

Infino's answer: write a table, invert it in the background

Infino treats those dials as the engine's job, not yours — and the architecture is built for it.

You write to Infino like a normal table: rows, columns, your vectors sitting next to your metadata. Ingest is cheap and append-only — no index to pre-build, no reindex window, no write stalls.

Then, in the background, Infino inverts that row-ordered data into a globally distance-aligned layout — vectors reorganized by similarity across the entire corpus, so a query reads only the neighborhood that matters.

That background pass is where the tuning happens. As it builds the global structure, Infino measures your actual data — how vectors cluster, how well the coarse code predicts the true nearest neighbor — and stamps a serving plan: partition count, probe width, and rerank budget calibrated to hit your recall target at the lowest cost. At query time it adapts again: probing narrow when the answer concentrates in one neighborhood, wider when it doesn't, and keeping the rerank budget honest as it widens.

How we measured it

On Cohere-10M (10M × 768-d embeddings, a standard vector benchmark), Infino's default adaptive serving — no manual n_cent, nprobe, or rerank_mult, nothing tuned by hand — returns recall@10 ≈ 0.99. The engine calibrated the probe width to the data and found that a narrow search already clears the bar — so it hits that recall without over-probing, spending compute only where it actually helps.

That's the point: the same engine would probe wider on a harder corpus where a narrow search wouldn't clear the bar — because the width is stamped from the data, not from a config you guessed.

What "self-driving" actually means

That's the difference between a database that's easier to tune and one that tunes itself. Infino owns the write path, the index, and the cost/recall trade-off end to end — so you get row-database-simple writes and expert-tuned ANN reads, and spend compute only where recall actually needs it.

And this is just the engine. Infino's serverless offering takes it further — capacity, tiering, and scaling automated end to end.

Inside a Parquet superfile

Ashish Mishra — Tue, 28 Jul 2026 00:00:00 +0000

Jul 28, 2026 · engineering

An Infino superfile is both a Parquet table and a search artifact. A standard Parquet reader sees columns and rows. Infino reads those same bytes plus an inverted index for BM25 and a vector index for nearest-neighbor search.

There is no sidecar to coordinate. The data and both indexes become visible together when the file is committed.

Why use Parquet at all?

Parquet is built for column scans, not posting-list jumps or vector probes. We considered three places to put the search index:

three-designs.txt

sidecar
  parquet data ──sync──▶ search cluster / vector store
  two copies · two consistency models

new format
  columns + search indexes in a search-first file
  one copy · dedicated readers

index inside parquet
  row groups + search regions + parquet footer
  one copy · standard parquet columns

A sidecar gives each engine its preferred layout, but every write creates a synchronization boundary. A new format keeps one copy, but makes that engine responsible for every reader and connector the data needs elsewhere.

Embedding the index accepts Parquet’s access-pattern constraints in exchange for keeping one immutable file and its existing columnar contract. The rest of the article is how that trade works at the byte level, and where it still hurts.

The byte layout

Parquet row groups come first and the Parquet footer remains last. The search regions occupy the space between them:

superfile.sf.parquet

byte 0
┌────────┬────────────────────┬────────────────┬──────────────┬────────┬────────┐
│ PAR1 │ parquet row groups │ full-text index│ vector index │ footer │ PAR1 │
└────────┴────────────────────┴────────────────┴──────────────┴────────┴────────┘
                                                          end of file

footer metadata:
  full-text offset + length + column config
  vector offset + length + column config

The row groups are written by the normal Arrow Parquet writer. Their byte offsets never change. The footer is an ordinary Parquet footer with additional namespaced metadata that tells Infino where each optional search region begins and ends.

Build the body, then rewrite the footer

The index offsets are unknown until the row groups have been encoded, so the builder cannot provide them as ordinary writer properties up front.

The build runs in three steps:

Write a complete Parquet file with the user columns and row groups.
Remove the original footer and append the full-text and vector regions.
Emit a new standard footer carrying the original row-group metadata plus the index offsets.

build-order.txt

standard parquet
      │
      ├── keep row-group bytes and metadata
      ├── append full-text region
      ├── append vector region
      └── write footer with final offsets

result: one immutable file

Appending the regions does not move the row groups, because they were already before the old footer. The replacement footer continues to point at the same column bytes.

Why a normal reader still works

A Parquet reader starts from the trailing PAR1, reads the footer, and follows the row-group offsets named there. The search regions sit outside those ranges. Metadata keys the reader does not recognize are ignored.

That is enough for a normal reader to project columns, apply predicates, and run SQL without knowing that BM25 or vector structures are present. Infino opens the same footer, recognizes its metadata namespace, and also addresses the embedded regions.

The compatibility guarantee covers reads. A generic writer emits the columns it understood and a new footer, so the search regions disappear. The result remains valid Parquet and requires reindexing before Infino can search it.

Parquet’s point-lookup tax

A Parquet reader decodes a page to retrieve one value. That is efficient for adjacent rows and wasteful when search needs to resolve one result by id.

Most columns use 1 MiB pages. On the corpus we measured, one page covered roughly 65,000 rows. The id column instead uses 8 KiB pages, roughly 512 rows, so resolving a shortlist does not decode a megabyte per hit.

Making every column use small pages was worse. On 320,000-row segments at k=10, full-row reads stayed flat while the id-plus-score path became about 8× slower. The reader had many more pages to plan over. Small pages belong on the point-lookup column, not everywhere.

Inside the full-text region

The full-text index contains a term dictionary, posting lists, document-length data, and per-block score bounds.

The dictionary is a finite-state transducer that maps each term to its posting list. Document ids are delta- and bit-packed in fixed-size blocks together with term frequencies. Each block records the maximum BM25 contribution any document in that block can make, allowing the query engine to skip blocks that cannot enter the current top-k.

Ranked BM25 and boolean token matching share those posting lists. Prefix lookup starts from the ordered term dictionary. Exact phrase matching first intersects the member tokens, then verifies the candidate against the stored string value; it does not require a second text copy.

Inside the vector region

The durable object-storage path uses OPANN with Sq16 data. Vectors are organized into contiguous regions so a query can select relevant clusters, fetch bounded ranges, score their quantized vectors, and rerank a shortlist without downloading the full column.

When a working set is pinned in RAM, Infino can use HNSW over the resident vectors instead. That is a serving choice above the durable file: the Parquet superfile remains the source of truth, and a cold worker can fall back to the object-storage search path.

This split follows the hardware. HNSW is effective when graph hops resolve in memory. Object storage favors a small number of planned, contiguous reads, which is what the clustered OPANN layout provides.

Checksums and range reads

Each index region is checksummed, so corruption is detected before a query trusts its contents. The Parquet footer locates the top-level full-text and vector regions; the headers and directories inside those regions locate the term dictionary, posting blocks, vector clusters, and rerank data.

An eager reader can materialize the complete file. An object-store reader instead opens a byte source and fetches the footer and region directories first. A BM25 query then reads the posting lists for its terms. A vector query reads the selected cluster ranges. The rest of the object stays in the bucket.

From one file to one table

A table is a manifest over many immutable superfiles. Before opening any file, the manifest uses scalar ranges, term-presence summaries, and vector routing data to reject files that cannot contribute to the query. The surviving files execute their local search, and the table layer merges the results.

The file and table layers therefore solve different problems: the superfile keeps data and indexes atomic; the manifest makes many superfiles behave as one snapshot.

The format is intentionally simple at its boundary: Parquet row groups, opaque search regions, and a standard footer describing both. Infino adds the search access paths Parquet lacks while keeping data and indexes in one immutable file.

Cutting a ten-term BM25 OR from 74.6 ms to 20.0 ms

Ashish Mishra — Tue, 23 Jun 2026 00:00:00 +0000

Jun 23, 2026 · engineering

On a warm 10-million-document real-cloud benchmark, a ten-term BM25 OR took 74.6 ms. Five terms took 24.4 ms; three common terms took 14.3 ms. The postings were not expensive to decode. The time was going into coordinating the posting lists one candidate document at a time.

10m-common-term-or.txt

query shape maxscore windowed reduction
3 similar terms 14.3 ms 8.2 ms 43%
5 similar terms 24.4 ms 12.0 ms 51%
10 similar terms 74.6 ms 20.0 ms 73%

Azure Standard_D8s_v7 · 8 vCPU / 31 GiB · warm p50 · baseline a7528d2 · windowed scorer shipped in PR #280

For faster profiling iterations, we repeated the same query family on a one-million-document in-memory tier and timed decode separately from merge and score. Decode was 3.3% of the ten-term query. Merge and score were 96.7%. Optimizing compression or I/O would not fix it.

Why MaxScore stopped skipping

Block-Max MaxScore is fast when one term has a meaningfully different upper bound. The low-scoring lists become non-essential, and whole blocks can be skipped once they cannot beat the current top-k threshold.

Common-term ORs have nearly uniform upper bounds. No term dominates, the essential set stays large, and block maxima are close to term maxima. The skip tests barely fire. MaxScore degrades into scoring almost the entire union while repeatedly finding the smallest document id across all active cursors, chasing their pointers, and touching the top-k heap.

Decode time stayed roughly flat at 0.4 to 1.0 ms as the query grew. The per-document merge grew from 5.7 ms to 10 ms to 30.5 ms. That was the part to remove.

Score one flat window at a time

The replacement processes a 4096-document range at a time. Each term walks its postings in that range and adds its BM25 contribution to a flat score array. A presence bitset records which document ids appeared. After every term has contributed, the scorer drains the set bits in document order into the existing top-k heap, then reuses the buffers for the next range.

The amount of scoring is unchanged when nothing can be skipped. What disappears is the per-document N-way cursor merge. Term-at-a-time writes are sequential and cache-friendly; the ordered drain happens once per window.

The lists advance together. Hits land in a 4096-document window, drain into the heap, then the next window starts.

At 10 million documents, the ten-term query fell from 74.6 ms to 20.0 ms , a 73% reduction. Five terms fell from 24.4 ms to 12.0 ms; three common terms fell from 14.3 ms to 8.2 ms. A brute-force oracle confirmed that the top-k results stayed identical across term counts, negation, and k.

Dispatch by query shape

Windowing complements MaxScore. The dispatcher chooses between them from the shape of the upper bounds:

Single-term queries keep the block-skipping path.
Two-term ORs with a rare anchor and bounded top-k use the WAND/BMW path, where the short posting list provides a useful pivot.
AND queries keep leapfrog intersection.
Multi-term ORs with a dominant upper bound stay on Block-Max MaxScore, where pruning is effective.
Multi-term ORs that pass the union-bound gates use the windowed scorer, where per-document coordination is the larger cost.

Each scorer wins on a different disjunction shape. MaxScore wins when bounds create skips; a flat window wins when the union must be scored anyway.

The 3 to 10 regression from running decode on tokio’s I/O pool

Ashish Mishra — Tue, 19 May 2026 00:00:00 +0000

May 19, 2026 · engineering

Infino’s public API is synchronous, while its storage I/O is asynchronous. A query may look like one blocking function call to the caller, but underneath it drives object-store reads through tokio and CPU-heavy decode through a sync-to-async bridge.

The wrong pool

Cold vector search had been about 1.1 s. After decode moved onto the tokio runtime that issued the storage reads, the same work took 3.7 s to 11 s : a 3× to 10× regression.

The cause was scheduler starvation. Tokio tasks yield at await points; decode has none. Once a worker began decoding a fetched block, it held that thread until the CPU work finished. Other tasks assigned to the same worker could not issue their next object-store reads, so CPU work serialized the I/O it was supposed to consume.

Tokio for I/O, rayon for CPU

The bridge now hands decode to rayon and awaits the result through a oneshot channel. Tokio’s workers remain available to schedule reads, while rayon’s bounded pool handles the data-parallel work. No thread acts as both an async-runtime driver and a rayon worker.

spawn_blocking would move work off the async workers, but onto tokio’s separate blocking pool. Decode is short, bounded, and data-parallel, so rayon gives it the pool and work-stealing policy we want without creating a second async runtime.

An async task hands CPU work to rayon and awaits a oneshot channel, so the tokio I/O threads never block on decode.

The range-fetch bridge has three caller contexts

Lazy object-store range reads can be requested from a multi-thread tokio runtime, a current-thread test runtime, or a plain rayon/CLI thread with no runtime. That Send + 'static bridge takes a different path in each case:

With an ambient multi-thread runtime, block_in_place tells the scheduler that this thread will block, allowing it to move other async work elsewhere.
With no ambient runtime, a current-thread runtime polls the storage futures inline. There is no worker-thread handoff to coordinate.
With an ambient current-thread runtime, block_in_place is illegal. The bridge runs the operation on a dedicated worker thread instead of blocking the only runtime thread.

Other sync-to-async entry points use a narrower contract. The main query path receives Infino’s shared multi-thread I/O runtime and always drives the future there; an ambient runtime only determines whether block_in_place is needed. The three-way bridge above is reserved for range-fetch futures that can safely move to a dedicated thread.

The range-fetch bridge picks one of three paths depending on which tokio runtime, if any, is already running.

The second regression

An earlier version of the generic no-runtime path had another avoidable handoff. It reused one multi-thread runtime pinned to a single worker. Reuse sounded cheaper than building a runtime per call, but every block_on poll had to coordinate with that worker. A current-thread runtime polls the same future inline.

The shared runtime added 6 to 17 percent to multi-term full-text queries at 10 million documents, with the penalty increasing as the query fanned out. Single-term queries were flat. Building a current-thread runtime for the no-runtime case removed that coordination cost.

The rule is now explicit: tokio schedules object-store I/O, rayon performs decode, and the bridge adapts to the caller without mixing those responsibilities.

The three costs of a cold query on S3

Ashish Mishra — Tue, 28 Apr 2026 00:00:00 +0000

Apr 28, 2026 · engineering

Object storage holds Infino’s durable copy; compute and local cache can disappear. The first query on a new worker therefore has to discover each file’s layout, open enough connections to read many files concurrently, and populate cache without downloading the whole dataset.

On a 10-million-document table in us-east-1, opening one cold superfile took about 164 ms ; a fresh table with a couple hundred files took about 556 ms. Those numbers reflect three categories of cold-path work.

These delays overlap: file opens fan out, cache state differs by file, and connection setup runs concurrently. Together they are the three places a first query can be forced to wait.

1. Discover the file in one GET

Parquet puts its footer at the end of the object. On local disk that is one seek. A textbook S3 reader sends a HEAD request to learn the object size, then a range GET for the tail: two round trips before it knows where the columns and indexes live.

S3 suffix ranges remove the HEAD. A request for bytes=-N returns the tail and reports the full object size in Content-Range. Infino starts with a 64 KiB suffix, enough for the common footer. A targeted second GET runs only when the footer begins before that window.

One suffix-range GET returns the tail and the object size. A second GET fires only when the footer started before the 64 KiB window.

In-region, the suffix request removed about 25 to 50 ms from a cold open. When the manifest already records exact ranges, the footer and index metadata can be fetched in one parallel batch. Small superfiles can carry their entire open window in that read.

Azure Blob does not accept the same suffix-range request, so a standalone Azure open still uses HEAD plus a bounded read. A table manifest already records the object size, which removes that extra request from the normal query path.

2. Keep connections ready for fan-out

A query may open hundreds of superfiles concurrently. Saving one metadata request does little if every range read first establishes TCP and TLS. Connection setup adds another serialized round trip and becomes visible in the tail.

The pool keeps up to 1024 idle connections per host so fan-out can reuse sockets. S3 connections expire from the pool after 10 seconds , below S3’s roughly 20-second server-side idle close; a 5-second connect timeout prevents one stalled handshake from holding the query. Azure uses a 90-second pool timeout to stay below its longer server-side window.

3. Drop RAM without dropping the file

After a superfile is cached locally, Infino memory-maps it. That consumes two different budgets: resident pages in RAM and the file on local disk. An idle-threshold sweep (and a budget sweep when a memory cap is configured) uses MADV_DONTNEED to drop resident pages while retaining the disk file, so the next access faults from local storage rather than S3.

Pinning every file in RAM would remove that fault, but it would also recreate the always-on storage cost this architecture is meant to avoid. The disk copy is the middle state: slower than resident pages, faster than another object-store read, and disposable.

The cache files are immutable and mapped read-only. POSIX also keeps an existing mapping valid after its file is unlinked, until the last reference disappears. A running query can therefore finish while the cache evicts the file from its directory.

Two budgets. Dropping pages from RAM leaves the local file, so the next touch skips the download.

Where the design stops working

The cold-open cost is paid when a worker encounters files it has not seen before. Repeated queries reuse the open metadata, connections, disk files, and resident pages. The architecture works when that cache has time to become useful.

On the same 10-million-document table, a cold analytic that touched every segment ran as high as eight seconds. A workload made entirely of first-touch scans is the wrong fit. For tables that should become warm, prefetch runs at a bounded fan-out of 8 , filling the local cache without sending every range request at once.

Object storage becomes practical as the durable database only when the query path bounds its requests, reuses its sockets, and treats local disk as a disposable cache.