---
title: "NEAREST BY Join: Scaling Vector Search in Databricks Runtime"
description: "Batch vector search workloads, such as entity resolution and semantic tagging, require high throughput across millions of queries against billions of vectors, unlike real-time serving. Databricks built NEAREST BY, a first-class top-k ranking join syntax, directly into its runtime engine rather than federating to an external endpoint. The feature is implemented as a logical join that the optimizer lowers onto standard relational operators, enabling fault tolerance and elastic parallelism. A fused Photon operator collapses the plan’s middle section with a blocked GEMM kernel designed to approach peak arithmetic intensity for vector scoring. An optional IVF index, stored as an ordinary liquid-clustered Delta table, allows approximate queries to probe only relevant clusters while preserving the same primitives and correctness guarantees."
---

# NEAREST BY Join: Scaling Vector Search in Databricks Runtime

[Databricks](https://yomu.fyi/company/databricks) · Zero Qu (Zero), Alexis Schlomer, Akash Nayar, Yingyi Bu, Sergei Tsarev · Oct 5, 2026

**Type:** Explainer

## Summary

Batch vector search workloads, such as entity resolution and semantic tagging, require high throughput across millions of queries against billions of vectors, unlike real-time serving. Databricks built NEAREST BY, a first-class top-k ranking join syntax, directly into its runtime engine rather than federating to an external endpoint. The feature is implemented as a logical join that the optimizer lowers onto standard relational operators, enabling fault tolerance and elastic parallelism. A fused Photon operator collapses the plan’s middle section with a blocked GEMM kernel designed to approach peak arithmetic intensity for vector scoring. An optional IVF index, stored as an ordinary liquid-clustered Delta table, allows approximate queries to probe only relevant clusters while preserving the same primitives and correctness guarantees.

## Context

Batch vector search workloads—entity resolution, deduplication, semantic tagging—require high throughput across millions of queries against billions of vectors, but existing real-time serving endpoints are not designed for batch jobs. Databricks needed a solution that integrates with its runtime engine to leverage elastic parallelism, fault tolerance, and peak arithmetic performance.

## Approach / What changed

Databricks introduced NEAREST BY, a SQL join syntax that expresses batch vector search as a top-k ranking join. The implementation uses a query rewrite onto standard relational operators and native Photon SIMD functions, plus a fused GEMM kernel for high arithmetic intensity. An optional IVF index, stored as a liquid-clustered Delta table, provides approximate search with the same engine primitives.

## Takeaways

- NEAREST BY treats batch vector search as a top-k ranking join, making it a first-class relational operation that integrates with the runtime engine’s existing shuffle, spill, and retry mechanisms.
- The fused Photon operator uses a blocked GEMM kernel to increase arithmetic intensity, moving the bottleneck from memory bandwidth to compute and achieving better throughput for scoring.
- The optional IVF index is stored as an ordinary Delta table with liquid clustering, so approximate queries probe only relevant clusters while the index itself requires no separate infrastructure.

## FAQ

### How does NEAREST BY handle fault tolerance in batch jobs?

NEAREST BY leverages the Databricks runtime engine's fault tolerance properties: tasks retry automatically on failure, memory pressure spills to disk, and the job can scale resources elastically. No client-side retry loops or rate limiting are needed.

### Can the approximate (APPROX) query change results if an index is created or dropped?

No. Only queries that explicitly specify APPROX consent to approximation. Creating or dropping an index never silently changes exact query results, maintaining a clear semantic contract.

### How is the IVF index stored and refreshed?

The index is stored as an ordinary Delta table, liquid-clustered by centroid ID. Refresh is transactional and incremental; files added since the last refresh are brute-forced in a compensation branch, so a stale index may prune less but never reduces search quality.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark), [Databricks](https://yomu.fyi/topic/databricks), [SQL](https://yomu.fyi/topic/sql), [Vector Search](https://yomu.fyi/topic/vector-search)

- Source: [Databricks](https://www.databricks.com/blog/nearest-join-scaling-vector-search-databricks-runtime)
- Source URL: https://www.databricks.com/blog/nearest-join-scaling-vector-search-databricks-runtime
- Ingested by Yomu: 2026-10-05T21:00:08.021Z

[Read original post](https://www.databricks.com/blog/nearest-join-scaling-vector-search-databricks-runtime)