Databricks Runtime Introduces NEAREST BY: Native SQL Vector Search at Scale

Vector search has become a foundational building block for retrieval-augmented generation (RAG), recommendation systems, semantic matching, and AI-powered data applications.

But there is an architectural detail that is easy to overlook: not every vector-search workload is an online serving problem.

Sometimes you need to match millions of query vectors against a large collection of embeddings in a scheduled job. You might be matching transactions to merchants, finding similar products, resolving duplicate records, or generating candidate pairs for a machine-learning pipeline.

In those cases, routing every query through a separate real-time vector-search endpoint can introduce an unnecessary dependency.

Databricks is addressing this problem with NEAREST BY, a native SQL join for batch vector search in Databricks Runtime. The implementation, described in a Databricks engineering article published on October 5, 2026, combines a fused Photon operator, a specialized General Matrix Multiply (GEMM) kernel, and an approximate nearest-neighbor index stored in a liquid-clustered Delta table.

The idea is straightforward: instead of treating every vector lookup as a remote service request, bring the search operation into the analytical query engine.

For data engineers and AI engineers, that could simplify an important class of workloads.

NEAREST BY on one page: query vectors and a Delta vector table meet in a SQL join, Photon's fused GEMM scores them, and APPROX or EXACT decides how the top-k is found.

Why batch vector search matters

Databricks describes a payments company that matches more than 100 million daily transactions against 140 million merchant embeddings for entity resolution.

Or picture a nightly job that compares millions of new product embeddings against an existing catalog.

The conventional architecture might look like this:

Query vectors
     |
     v
Batch orchestration
     |
     v
External vector-search endpoint
     |
     v
Results returned to Spark
     |
     v
Downstream transformations

This can work well, particularly when the application needs low-latency online retrieval. But for large batch workloads, it introduces a separate service, request orchestration, network overhead, and another component to monitor.

With NEAREST BY, the architecture can instead look like this:

Query vectors                 Base vectors
      |                             |
      v                             v
  Query table                Delta vector table
      |                             |
      +-------------+---------------+
                    |
                    v
             NEAREST BY JOIN
                    |
                    v
          Top-k matching vectors
                    |
                    v
          Downstream SQL / Spark

Databricks describes this as bringing batch vector search directly into the lakehouse, with no separate vector-serving system required for the supported query pattern: "Your Lakehouse is also your vector store."

The practical benefit is architectural simplicity: query vectors, indexed vectors, and downstream processing can participate in the same SQL workflow.

That doesn't eliminate every external vector database use case. Interactive applications with strict online latency requirements may still need a dedicated serving architecture, such as the online AI Search path we covered in our Databricks lakehouse breakdown. The important distinction is choosing the right search architecture for the workload.

What is NEAREST BY?

NEAREST BY extends SQL joins with a top-k nearest-neighbor operation.

For each row on the left side of a join, it finds up to k matching rows on the right, ranked by a distance or similarity expression.

The syntax supports two key choices:

• EXACT: returns the exact top-k results according to the ranking expression.
• APPROX: allows approximate search strategies to return a top-k set that approximates the exact ranking.

The ranking expression can use vector similarity or distance functions, including cosine similarity, L2 distance, and inner product. A query can therefore express nearest-neighbor retrieval as a relational operation rather than manually orchestrating a table-valued function or an external API.

The official NEAREST BY SQL reference documents the join semantics and supported ranking expressions.

A simplified example illustrates the idea:

SELECT
    q.query_id,
    b.item_id,
    vector_cosine_similarity(
        q.embedding,
        b.embedding
    ) AS similarity
FROM query_vectors AS q
JOIN base_vectors AS b
    APPROX NEAREST 10
    BY SIMILARITY vector_cosine_similarity(
        q.embedding,
        b.embedding
    );

This example illustrates the intended query shape; check the current Databricks SQL reference and your runtime version for exact syntax and supported expressions before deploying it.

The important point is that top-k matching becomes part of the SQL plan. The runtime can choose how to distribute the work and execute the ranking, instead of requiring the application to issue one network request per query vector.

---

This is an excerpt. The full article continues with:

• How Databricks makes batch vector search fast
• APPROX versus EXACT is an important contract
• What Databricks reports about scale
• What data and AI engineers should do
• Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.