Loading…
Topic
Databricks
222 posts about Databricks. Every summary links to the original.
- 222 posts
- 3 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
NEAREST BY Join: Scaling Vector Search in Databricks Runtime
Batch vector search workloads, such as entity resolution and semantic tagging, require high throughput across millions of queries against billions of vectors, unlike real-time serving. Databricks built NEAREST BY, a first-class top-k ranking join syntax, directly into its runtime engine rather than federating to an external endpoint. The feature is implemented as a logical join that the optimizer lowers onto standard relational operators, enabling fault tolerance and elastic parallelism. A fused Photon operator collapses the plan’s middle section with a blocked GEMM kernel designed to approach peak arithmetic intensity for vector scoring. An optional IVF index, stored as an ordinary liquid-clustered Delta table, allows approximate queries to probe only relevant clusters while preserving the same primitives and correctness guarantees.
How Genie Ontology powers product development at Databricks
General-purpose AI agents fail on business questions because they lack enterprise context such as authoritative tables, business rules, and trusted experts; the missing institutional knowledge is scattered across dashboards, documents, and people. Databricks’ Genie Ontology addresses this by continuously updating institutional knowledge from certified Unity Catalog concepts and automatically learned snippets from dashboards, notebooks, SQL queries, and connected apps. OntoRank ranks sources by authority using certification, usage, and authorship signals, so Genie grounds answers in the most trusted and relevant data. In a weekly adoption review, a product manager prompts Genie, which uses MCP to search Google Docs and Jira while running live SQL for fresh analysis. The resulting report includes citations, forecasting, scheduling, and even conversion into a shareable Genie Agent, all within permission boundaries.
How Genie One reshapes work for finance teams
Finance teams must interpret data to guide decisions, but answering follow-up questions often requires manual extraction and metric negotiation, delaying answers past leadership discussions. Genie One is a data-smart AI coworker grounded in business context such as approved metric definitions, fiscal periods, and entity rollups, using Genie Ontology to unify context. It supports workflows including budget variance analysis, cash flow forecasting, financial close management, and spend analytics, allowing plain-language queries like “Which entities have unreconciled intercompany balances above $100,000?” Users can review tables behind answers, schedule recurring reviews, and have results posted to chat with visualizations. Genie One respects each user’s existing permissions, so a corporate controller sees enterprise-wide items while a regional accountant sees only relevant entities.
In Capital Markets, the Buy Side Runs on NAV. Finance Protects the Fee.
Buy-side asset management finance departments face growing complexity from AI agents influencing valuation, redemption, and reconciliation workflows, while protecting net fee margin and NAV integrity remains their core mission. Databricks Genie One is presented as a data-smart AI coworker grounded in Genie Ontology, which captures business meaning and keeps definitions current as strategies and fee arrangements shift. Behind the assistant, Genie Agents automate overnight fund operations: ingesting custodian files, pricing positions, reconciling records, and producing a preliminary NAV for review. Every figure is traceable to its source, permissioned by role, and provable to investors or regulators, enabling finance teams to see forming risks early. The system does not make investment or valuation decisions; a human remains in the loop for every action.
Real-Time Retail Intelligence: Building E-Commerce Recommendations with Lakebase and AI Search on Databricks
The post presents a reference architecture for building a production-grade e-commerce recommendation system on Databricks, based on a real-world implementation for a leading fashion e-commerce platform in Asia serving over 1 million monthly active users across 100,000+ SKUs. The system ingests approximately 1,000 events per second via Lakeflow Connect’s Zerobus into Unity Catalog Delta tables within a medallion architecture. Serving uses two paths: Path A for pre-computed batch recommendations via key-value lookup from Lakebase online tables, and Path B for real-time session-aware scoring using AI Search and a LightGBM model in a 3-stage funnel. The architecture unifies ingestion, feature management, model training, vector retrieval, and serving on a single governed platform, with cold-start handling for new users and products. Expected outcomes include low 2-digit millisecond responses, daily-fresh recommendations, and unified governance via Unity Catalog.
Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables
Unity Catalog managed tables let you control where your data physically resides while keeping it in cloud storage you own, such as S3, ADLS, or GCS, unlike platforms that store tables in provider-controlled storage. You set managed storage locations at the metastore, catalog, or schema level, with the most specific level winning, and can change where new tables land using ALTER CATALOG or ALTER SCHEMA commands. Managed tables use open formats including Iceberg and Delta, and external engines such as Apache Spark, Flink, and Trino can read and write through Iceberg REST Catalog and Unity Catalog open APIs. When converting an external table to managed, Databricks copies the data and transaction log into the managed storage location you have set. This approach enables a single source of truth across analytics and AI workloads while satisfying compliance, cost allocation, and data residency requirements.
From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP
S&P Global Energy aimed to make its structured data estate—spanning LNG, Chemicals, Crude Oil, and other commodities—conversationally accessible via AI agents. The team evaluated approaches and chose Databricks Genie Agents, where subject matter experts curate one agent per dataset group without writing code. Each Genie Agent is automatically exposed as a managed MCP server, providing a governed, zero-deployment integration point. A FastMCP-based proxy composes multiple Genie MCP servers into composite endpoints for cross-domain questions. This architecture shortened time to market from months to days, with governance via Unity Catalog and an open MCP standard enabling both internal and external consumers.
Running open-Jev in SQL on Databricks
System One decision models such as Jev produce well-calibrated decisions from discrete options and are fast and cheap for large-scale applications, with potential use cases ranging from analyzing customer support transcripts to applying complex decisions at scale. Open-weight versions like SemIf-OpenJev are available, and Databricks enables serving them on governed data using serverless GPUs via AI Runtime, which provides on-demand GPU compute without managing infrastructure. The workflow is packaged in an importable notebook that downloads the model files, logs and registers the model using Express Deployments, and creates a GPU Model Serving endpoint automatically. The endpoint can be called from SQL via ai_query, which supports arbitrary custom model APIs, allowing structured request formats directly from SQL to return selected classifications and option probabilities as structured columns. The example classifies hotel reviews as good or bad, and users can import the notebook, select Serverless GPU, and run all to get started.
How I built agent-based security reviews on Databricks
Security reviews at Databricks relied on automation but still consumed expert time on predictable work, leaving less room for genuinely novel, high-risk architectures. The author built an agent-based layer on Databricks using Unity Catalog for governed tables containing standards, request data, and decisions; Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning; and Lakeflow Jobs for orchestrating notebook-based workflows on serverless compute. A conversational intake app built with Databricks Apps turns plain-language descriptions into structured requests and provides preliminary risk indications before a formal request is created. Seven focused agents each have a bounded responsibility—intake, risk assessment, requirements, specialized reviews, validation, workflow, and learning—ensuring behavior remains inspectable and testable. Routine requests that once waited days in the queue can now be completed in minutes, and security engineers spend more time on designs requiring genuine expertise.
How Concurrence governs clinical AI at a trillion-token scale with Unity Gateway
Concurrence processes 100.8 billion input tokens and 11.2 million LLM calls every 30 days, at an annualized rate of 1.2 trillion tokens. To govern these workflows, the company consolidates on Databricks with Lakebase for operational state, Unity Catalog for governance, and Unity Gateway for centralized AI access. Patient data is recorded as immutable events in Delta tables, from which a current patient state—called the world model—is computed while preserving provenance. All agents are tested against simulated patients before deployment; simulation traffic is approximately seven times production traffic, using immutable event history to replay states. Concurrence routes all coding-agent traffic through Unity Gateway, attributing each request to the engineer, and enforces compliance-gated model routing where HIPAA requirements are met first.
Unity Catalog Pages: a governed home for your business knowledge in Genie Ontology
Business concepts like active customer or completed trip are often defined differently across teams, scattered in Slack threads and spreadsheets, causing AI agents to guess confidently when answering questions. Databricks introduces Unity Catalog Pages as a governed home within Genie Ontology where data stewards with Genie Code define the authoritative meaning of those terms and entities. Pages complement existing semantics — metric views how to calculate, domains where to look, and certification what to trust — by capturing what each concept means. Genie Code can bootstrap Pages by extracting definitions from Confluence, Slack, Google Docs, or GitHub via MCP-connected tools, turning weeks of work into minutes. When Genie One uses a Page to ground an answer, the Page is cited as a clickable source so users can inspect the definition and verify the reasoning.
Database Branching: A Developer's Guide to Git-Style Workflows
Database branching creates isolated database environments from a known parent state using copy-on-write, where branches share unchanged data and only store modifications. For a large parent database like one that is 40 GB, two branches may require just 5.6 MB of additional storage instead of 80 GB with traditional full copies. Developers can test schema migrations like adding a NOT NULL column to a table with millions of rows against realistic data and constraints, CI pipelines provision per-pull-request branches for isolated tests, and AI agents experiment across short-lived environments without risking the parent database. Safe operation relies on protecting parent branches from writes and deletions, restricting sensitive data in disposable branches, automating creation and cleanup, and keeping environments reproducible through version-controlled configuration. Databricks Lakebase provides database branching, enabling teams to adopt Git-style workflows for development, CI/CD, and AI agent use cases.
The Genie One MCP is now Generally Available
Databricks has announced general availability of the Genie One Model Context Protocol (MCP) server, now a managed service within Unity Gateway with centralized governance, fine-grained policies, and audit logging across every invocation. The MCP provides a single governed interface for any AI agent to access structured and unstructured data from Genie One, grounded in governed business context from Genie Ontology, eliminating agent sprawl and inconsistent answers. It exposes tools for asking questions, getting query results, checking incremental progress, and steering responses, with an optional MCP App for real-time interactive visualizations and Genie Ontology citations. Connected agent clients automatically interpret domain semantics via Genie Ontology across relational data and document repositories without custom connectors. Use cases include enriching slide creation with richer data, bringing customer usage context to outreach agents, and integrating business definitions into developer workflows, as noted by GetYourGuide.
RADAR: Catch gray failures with anomaly detection
Gray failures are partial customer-facing problems that can leave health checks green while a subset of users quietly fails. RADAR (Reliability Anomaly Detection, Alerting, and Root-cause analysis) addresses this gap by collecting error counts and distinct affected users, segmented by error code and region, then applying the unsupervised streaming SPOT model. SPOT learns normal behavior from the previous 14 days using a single risk parameter, while downstream stages enrich, filter, deduplicate, and route alerts before providing anomaly details and AI/BI Genie-backed dashboards for investigation. Running RADAR internally reduced incident-discovery time by 95% with over 90% precision and no human needed to spot the pattern. The post also describes a Databricks scaffold that maps metrics, detection, alerting, and root-cause analysis to platform components and can be assembled by an AI agent from a metric, scaffold, and prompt.
Database for AI Agents: 5 Evaluation Criteria
The post defines a database for AI agents as infrastructure for short-term, episodic, procedural, and operational memory, rather than merely a place to retrieve documents. It proposes five evaluation criteria for moving agent workloads from prototypes to production: per-agent branch isolation, serverless scale-to-zero, hybrid search, ACID guarantees, and unified platform access. The technical rationale is that agents perform repeated reads and writes, resume tasks, share state concurrently, and need current data across vector, keyword, and metadata retrieval; zero-copy branches, sub-second wakeups, and single-query hybrid search address specific operational constraints. It concludes that a database missing more than one minimum bar becomes a production risk, while standard Postgres alone supplies ACID but not branching, scale-to-zero, or unified operational and analytical access.
Google Cloud Use Cases: How Enterprises Run Data & AI on GCP with Databricks
The guide presents Google Cloud as a unified platform for enterprise data engineering, analytics, machine learning, and generative AI, with managed services intended to reduce infrastructure and integration work. It focuses on practical architectures, including medallion ETL pipelines that store Bronze, Silver, and Gold Delta Lake tables in Google Cloud Storage and expose Gold data to BigQuery through an optimized connector. For streaming workloads, Cloud Pub/Sub supplies ingestion while Apache Spark Structured Streaming processes continuous micro-batches; together with Delta Lake ACID transactions, this supports exactly-once results despite duplicates or out-of-order messages. The text also describes Cloud Run and Cloud Functions for containerized or event-driven workloads, and identifies retrieval-augmented generation applications on Vertex AI as a common starting point for generative AI. It concludes that integrated service combinations, rather than isolated tools, are the central strategic advantage for organizations building evolving data and AI platforms on Google Cloud.
Databricks recognized as a Gartner® Peer Insights™ Customers' Choice for Analytics and BI
Databricks says it was recognized as a Customers’ Choice in Gartner Peer Insights’ Voice of the Customer for Analytics and Business Intelligence Platforms, based on verified reviews from customers using platforms in production. The post reports a 4.8 out of 5 overall rating and 94% willingness to recommend, while noting that the designation reflects customer adoption and overall experience meeting or exceeding market averages. It presents Databricks AI/BI, AI/BI Dashboards, Genie Spaces, and Genie as a unified architecture combining governed live data, AI, business intelligence, and Unity Catalog controls without moving or duplicating datasets. The described use cases include conversational exploration, dashboard and visualization development, consolidating fragmented reporting, and expanding governed analytics access, with the company positioning these capabilities as a foundation for agentic analytics.
How Databricks Helps Baseball Teams Gain an Edge with Data & AI
Baseball teams can use Databricks to turn Statcast and team-derived pitch-level data into game-day plans for hitters, bullpen changes, pinch-hit choices, advance scouting, and roster decisions. Genie, on Unity Catalog, queries governed Delta tables and reusable Unity Catalog SQL functions for tendencies by count, handedness, pitch type, location, and base-runner state. A Multi-Agent Supervisor built with Agent Bricks and deployed on Model Serving simulates matchup pockets, compares reliever arsenals with hitter profiles, and produces explanations that staff distill into printed decision cards. For advance work, AI Search indexes pitcher and hitter embeddings to find similar arms, while Lakebase Postgres keeps reports, tags, and decisions close to the source data for shared operational memory.
Building an A/B testing analysis framework for mobile gaming on Databricks
HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.
Fizz accelerates ecommerce analytics with Databricks SQL
Fizz.hu, an ecommerce marketplace backed by OTP Group, outgrew a Microsoft SQL Server and Power BI setup that relied on daily batch loads and separate virtual-machine Python workloads. The company migrated about 50 tables and several stored procedures to Databricks SQL, recreated core views, and moved to a lakehouse architecture using an MVP-first approach rather than redesigning everything. Completed in three months, the migration switched off SQL Server while keeping Power BI reports running on the new engine. Nightly processing fell from three to four hours to roughly 90 minutes, reports became consistently available by 4:30 a.m., Power BI refreshes dropped about 50%, and gigabyte-scale exports finish in minutes. Databricks now unifies SQL, Python, machine learning, governance work, and Hungarian-language Genie self-service, with about 20 users reclaiming roughly 20% of an analyst’s time.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.