Loading post…
DatabricksZero Qu (Zero), Alexis Schlomer, Akash Nayar, Yingyi Bu, Sergei TsarevExplainer
NEAREST BY Join: Scaling Vector Search in Databricks Runtime
Summary
Batch vector search workloads, such as entity resolution and semantic tagging, require high throughput across millions of queries against billions of vectors, unlike real-time serving. Databricks built NEAREST BY, a first-class top-k ranking join syntax, directly into its runtime engine rather than federating to an external endpoint. The feature is implemented as a logical join that the optimizer lowers onto standard relational operators, enabling fault tolerance and elastic parallelism. A fused Photon operator collapses the plan’s middle section with a blocked GEMM kernel designed to approach peak arithmetic intensity for vector scoring. An optional IVF index, stored as an ordinary liquid-clustered Delta table, allows approximate queries to probe only relevant clusters while preserving the same primitives and correctness guarantees.
Context
Batch vector search workloads—entity resolution, deduplication, semantic tagging—require high throughput across millions of queries against billions of vectors, but existing real-time serving endpoints are not designed for batch jobs. Databricks needed a solution that integrates with its runtime engine to leverage elastic parallelism, fault tolerance, and peak arithmetic performance.
Approach / What changed
Databricks introduced NEAREST BY, a SQL join syntax that expresses batch vector search as a top-k ranking join. The implementation uses a query rewrite onto standard relational operators and native Photon SIMD functions, plus a fused GEMM kernel for high arithmetic intensity. An optional IVF index, stored as a liquid-clustered Delta table, provides approximate search with the same engine primitives.
Takeaways
NEAREST BY treats batch vector search as a top-k ranking join, making it a first-class relational operation that integrates with the runtime engine’s existing shuffle, spill, and retry mechanisms.
The fused Photon operator uses a blocked GEMM kernel to increase arithmetic intensity, moving the bottleneck from memory bandwidth to compute and achieving better throughput for scoring.
The optional IVF index is stored as an ordinary Delta table with liquid clustering, so approximate queries probe only relevant clusters while the index itself requires no separate infrastructure.
FAQ
How does NEAREST BY handle fault tolerance in batch jobs?
NEAREST BY leverages the Databricks runtime engine's fault tolerance properties: tasks retry automatically on failure, memory pressure spills to disk, and the job can scale resources elastically. No client-side retry loops or rate limiting are needed.
Can the approximate (APPROX) query change results if an index is created or dropped?
No. Only queries that explicitly specify APPROX consent to approximation. Creating or dropping an index never silently changes exact query results, maintaining a clear semantic contract.
How is the IVF index stored and refreshed?
The index is stored as an ordinary Delta table, liquid-clustered by centroid ID. Refresh is transactional and incremental; files added since the last refresh are brute-forced in a compensation branch, so a stale index may prune less but never reduces search quality.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.