---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Data Mesh vs. Data Fabric: Key Differences and How the Lakehouse Resolves the Debate](https://yomu.fyi/post/data-mesh-vs-data-fabric-key-differences-and-how-the-lakehouse-resolve.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 25, 2026

Data mesh and data fabric address different constraints in distributed data architecture: mesh changes ownership and accountability, while fabric automates integration and governance. A data fabric uses active metadata, machine learning, centralized policy enforcement, and virtualized access to discover, govern, and connect data across heterogeneous environments with less data movement. A data mesh assigns domain teams responsibility for data products, quality, reliability, and lifecycle management, supported by self-serve infrastructure and federated computational governance. The article presents a lakehouse as a substrate where domains publish products while capabilities such as Unity Catalog, Delta Sharing, and Lakehouse Federation provide metadata, governance, and sharing. Its conclusion is that the approaches are complementary: organizations should identify whether organizational bottlenecks or technical fragmentation is the dominant constraint, then consider a hybrid.


### [Modernizing SQL ETL in Lakehouse with Declarative Patterns](https://yomu.fyi/post/modernizing-sql-etl-in-lakehouse-with-declarative-patterns.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matt Jones, Shanelle Roman
- Published: Aug 25, 2026

Databricks is extending declarative ETL patterns into the Lakehouse SQL Editor so SQL practitioners can define recurring transformations without a dedicated pipelines-oriented environment. The initial operations cover append-only updates, change data capture through AUTO CDC, and targeted batch overwrites through REPLACE WHERE; flows can be scheduled, triggered by upstream updates, run on demand, or orchestrated through SQL tasks in Jobs. APPEND tracks processed source data, AUTO CDC handles keys, sequencing, deletes, and SCD Type 1 or Type 2 results, while Enzyme can incrementally process changed data for REPLACE WHERE. In Lakehouse benchmark testing, Enzyme-powered REPLACE WHERE ran 3.4x faster and 2.5x cheaper than traditional REPLACE WHERE, while users can retain procedural SQL and later move to the Lakeflow Pipelines Editor for multi-file development, dependencies, validation, and deployment.


### [Open Table Formats Explained: Iceberg vs. Delta vs. Hudi](https://yomu.fyi/post/open-table-formats-explained-iceberg-vs-delta-vs-hudi.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 25, 2026

Open table formats add metadata layers over Parquet or ORC files in object storage, enabling ACID transactions, schema evolution, time travel, and database-like consistency for data lakes. Apache Iceberg uses manifests and manifest lists in a metadata tree, supports partition evolution without rewriting existing files, and creates immutable snapshots for auditing, rollback, and reproducible queries. Delta records atomic writes in an ordered transaction log with checkpoints and is closely associated with Spark, while Hudi uses record-level indexes for frequent updates, change-data-capture, and near-real-time ingestion. All three share core capabilities but differ in partition evolution, update and delete strategies, and multi-engine support; Parquet and ORC remain file formats rather than table formats. The overview recommends scheduled compaction, metadata maintenance, health monitoring, and a production-volume proof of concept when selecting a format.


### [Run, debug, and scale Databricks workloads from your local IDE](https://yomu.fyi/post/run-debug-and-scale-databricks-workloads-from-your-local-ide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Tanishq Maheshwari, Matt Jones
- Published: Aug 24, 2026

Databricks introduces updated IDE and CLI integrations that connect VS Code, Cursor, or a terminal directly to Databricks compute, workspace, and Unity Catalog. Existing Databricks Connect supported local Spark development, but remote non-Spark workloads and dependency synchronization remained pain points. The new SSH tunnel supports interactive execution and debugging of Python and SQL workloads on Serverless, AI Runtime, and dedicated clusters, while keeping files and dependencies aligned with Databricks Runtime and the workspace. CLI flags can select serverless, GPU-backed AI Runtime, or a dedicated cluster, and can start the tunnel in VS Code or Cursor; additional options configure base environments and usage policies, while the IDE can browse Unity Catalog assets.


### [How Databricks Uses AI to Accelerate Incident Investigation](https://yomu.fyi/post/how-databricks-uses-ai-to-accelerate-incident-investigation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Avijeet Gupta, Bhuban Seth, Kusum Madarasu
- Published: Aug 24, 2026

Databricks describes AI SRE, an AI-powered debugging agent for operating hundreds of microservices across more than 1,500 Kubernetes clusters, 70+ regions, and three clouds. It addresses incident investigations by assembling context from platform health, service telemetry, changes, dependencies, and team-specific runbooks, with automatic triage followed by interactive, natural-language investigation. The platform uses layered primitives, purpose-built APIs, and a core orchestration engine; team-owned agentic runbooks encode domain procedures, while controlled access and evidence links support auditing. The post reports support for more than 150 teams, 250+ weekly active users, and over 2,000 daily investigations, with users saving several hours of debugging time.


### [Relational vs Non-Relational Database: Choosing the Right Data Store](https://yomu.fyi/post/relational-vs-non-relational-database-choosing-the-right-data-store.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 24, 2026

Selecting between relational and non-relational databases requires matching the data store to requirements for structure, consistency, scale, and query complexity. Relational systems organize data in predefined tables, enforce schemas and keys, and provide ACID transactions, making them suited to structured, transactional workloads and complex joins. Non-relational systems support documents, key-value pairs, graphs, or wide-column structures, often using flexible schemas and schema-on-read while scaling horizontally and accepting eventual consistency for throughput and availability. The trade-offs include vertical versus horizontal scaling, SQL versus database-specific query languages, and database-enforced versus application-level integrity. The guidance recommends documenting workload needs, prototyping with realistic volumes and failure scenarios, and considering polyglot persistence when different workloads require specialized systems.


### [Transactional Vs Analytical Database: Choosing OLTP, OLAP, or Hybrid](https://yomu.fyi/post/transactional-vs-analytical-database-choosing-oltp-olap-or-hybrid.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 24, 2026

Transactional and analytical databases serve different workloads: OLTP handles high-volume, real-time reads and writes with ACID guarantees, while OLAP supports complex queries over large historical datasets for reporting and business intelligence. The comparison focuses on trade-offs involving latency, concurrency, data freshness, schema design, transaction size, and storage format. Row-oriented storage and normalized schemas support precise operational updates, whereas column-oriented storage and denormalized designs improve compression, scans, and aggregations across millions of rows. The source describes transactional systems as foundational for applications such as banking, e-commerce, healthcare, and reservations, while analytical systems support dashboards, forecasting, and ad-hoc analysis. Its recommended architecture is to run both systems independently and connect them with Change Data Capture, balancing operational reliability with analytical throughput.


### [Connecting retail demand planning to campaign and store execution](https://yomu.fyi/post/connecting-retail-demand-planning-to-campaign-and-store-execution.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jack Yallop, Pavi Singh
- Published: Aug 21, 2026

Retail and consumer goods teams often split performance review, demand planning, marketing activation, store execution, and measurement across disconnected systems, making it difficult to connect analysis with results. The Databricks demo presents a unified application built on data unification, governance, and intelligence, combining point-of-sale, loyalty, supply chain, media spend, inventory, and store-operations signals with role-specific access. Users can move from KPI and forecast-versus-actual analysis to SKU or store what-if planning, audience selection, campaign activation, replenishment, price-tag updates, assigned tasks, and post-campaign measurement. Genie supports natural-language querying, while specialized agents and a coordinating supervisor connect sales insights, demand planning, audience building, in-store operations, and measurement. The stated design goal is a governed, shared workflow that lets teams test decisions, execute them across stores, and evaluate outcomes without losing operational context.


### [Inbound Private Link now supports account-level Genie One, the account console, and custom URLs](https://yomu.fyi/post/inbound-private-link-now-supports-account-level-genie-one-the-account.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Robert Zhang, Manish Bansal, Chen He, Yankai Zhang
- Published: Aug 20, 2026

Databricks has expanded Inbound Private Link in Beta on AWS and Azure to cover account-level resources, including account-level Genie One, the account console, Governance Hub, and account-level APIs. The update also supports custom URLs and Managed Disaster Recovery stable URLs, while one shared General Access endpoint in any region can serve workspace and account-level UI and API resources, removing the need for one endpoint per region or workspace. These capabilities use context-based ingress, where account admins define allow and deny rules by caller identity, network source, and destination through account and workspace policies. Enabling private access to account-level resources requires registering and allowlisting a General Access endpoint with the account-policy and resolving the custom URL to that endpoint. Existing workspace-specific URLs, private access settings, and IP access lists continue to work in parallel, while service-direct and SCC relay endpoints retain per-region requirements.


### [Busting SQL Migration Myths: How New SQL Features Make Lift-and-Shift to Lakehouse Easier](https://yomu.fyi/post/busting-sql-migration-myths-how-new-sql-features-make-lift-and-shift-t.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Abhishek Dey, Laurent Léturgez
- Published: Aug 20, 2026

The post addresses the procedural core of warehouse-to-lakehouse migrations: legacy stored procedures that use temporary tables, cursors, control flow, multi-table updates, and transaction handling while remaining maintained by SQL teams. Using a composite daily-orders procedure based on an Oracle migration, it translates rather than rewrites the logic on Databricks, covering procedure registration in Unity Catalog, session-scoped temporary tables, cursor syntax, handlers, conditional scripting, and atomic transactions. Databricks SQL scripting supports cursors from Runtime 18.1, and BEGIN ATOMIC provides automatic commit or rollback plus row-level conflict detection; catalogManaged tables are required inside atomic blocks. The post says this mechanical translation can cut migration timelines by 50–75% for complex procedures, while preserving business logic and allowing the SQL team to maintain it, though the procedure still needs validation in a migration project.


### [Designing effective Genie Agents from a single prompt](https://yomu.fyi/post/designing-effective-genie-agents-from-a-single-prompt.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Megan Tupper
- Published: Aug 19, 2026

Generic agents may select the first relevant-looking revenue table rather than the finance-maintained source because they lack business context. Genie Agents address this by using a single prompt to assemble a domain-specific agent from governed Unity Catalog context, including structured data, documents, and files. The prompt identifies the desired outcome and relevant sources, while the agent can retrieve content from PDFs, Word documents, presentations, images, and tables under the requesting user’s permissions. The recommended process is to begin with one focused, repetitive job, test it against known questions or incidents, and expand only after measuring performance. Built-in benchmarks provide accuracy scores, while monitoring surfaces user questions and feedback that can reveal gaps in the curated context.


### [Databricks Document Intelligence: pushing the frontier for complex document extraction](https://yomu.fyi/post/databricks-document-intelligence-pushing-the-frontier-for-complex-docu.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jane Zhang, Arnav Singhvi, Ivan Zhou, Archika Dogra, Matthew Ding, Nihit Desai
- Published: Aug 18, 2026

Databricks introduces Precision Mode for its ai\_extract document extraction API, targeting enterprise workloads that challenge existing LLM and rules-based systems. The mode addresses long documents with cross-references, large nested outputs such as thousands of line items, and schemas requiring reasoning across financial statements or applying discounts. It combines custom-trained, efficient extraction models with an agent harness that decomposes jobs semantically, runs smaller tasks in parallel, preserves intermediate results, and reconciles them into a final structured output. On benchmarks covering roughly 9,000 documents, including files up to 2,000 pages and schemas with more than 300 deeply nested fields, Precision Mode reached 94.7% accuracy, seven points above the strongest frontier-model chunk-and-merge baseline, GPT-5.6 Sol; the source says it is now available through ai\_extract and the Information Extraction UI.


### [When it comes to Governance, Retailers need a control plane for context](https://yomu.fyi/post/when-it-comes-to-governance-retailers-need-a-control-plane-for-context.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: BJ Sullivan, Sarah Duffy, Tayo Olabumuyi, Logan Boyd
- Published: Aug 18, 2026

Retailers are moving from isolated generative-AI pilots to enterprise-wide adoption, where each store, headquarters, and digital workflow needs different data, permissions, models, and cost controls. The post argues that useful AI depends on governed business context, while fragmented tools can produce separate provider contracts, logs, permission models, cost centers, and answers to the same question. It proposes an open control plane that centralizes governance and model access without forcing employees into one application or vendor. The described Databricks design combines Unity Catalog for data and business context, Unity AI Gateway for routing and oversight, Foundation Model APIs for commercial, open-source, and custom models, and Genie for natural-language queries over governed data; the intended result is faster experimentation and deployment with visibility into access, spending, lineage, and auditability.


### [Evaluating AI Agents Live at the Grounded Reasoning Cup](https://yomu.fyi/post/evaluating-ai-agents-live-at-the-grounded-reasoning-cup.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks AI Research Team
- Published: Aug 18, 2026

The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.


### [How Databricks Feature Store serves features with sub-second freshness](https://yomu.fyi/post/how-databricks-feature-store-serves-features-with-sub-second-freshness.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ian Ackerman, Nick Joung, Abhay Bothra
- Published: Aug 17, 2026

Databricks Feature Store addresses the lag between batch-computed machine-learning features and decisions that depend on events from seconds earlier. It lets a feature definition drive both offline batch processing and online streaming pipelines, orchestrating Kafka ingestion, Spark Real-Time Mode (RTM), Lakebase, and Model Serving. For a rolling 10-minute transaction sum, RTM updates per entity state in local RocksDB, expires contributions per event, and writes revised values to Lakebase through a streaming JDBC sink. The described path reports end-to-end p99 200ms latency from Kafka arrival to online feature availability, while Model Serving retrieves declared dependencies automatically at inference. Tumbling and sliding windows remain options when fewer, cheaper updates are acceptable.


### [The prototyping tax is killing your AI roadmap](https://yomu.fyi/post/the-prototyping-tax-is-killing-your-ai-roadmap.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Navdeep Alam, Zhe Sun
- Published: Aug 17, 2026

The post argues that the “prototyping tax”—delay between an AI-powered idea and a working MVP—causes initiatives to lose momentum, often because organizational context is fragmented across teams, tools, and codebases. Traditional encapsulation and siloed domain knowledge can leave agents with syntax and contracts but not business intent, making production codebases harder to navigate than personal projects. The proposed response is agentic development grounded in business semantics and governed data, with alignment occurring through working builds while CI/CD, code review, and production rigor remain unchanged. The post recommends tracking time-to-prototype, first-pass acceptance rate, and PoC-to-production rate to distinguish faster demos from durable delivery. It cites a benchmark of 401 data tasks and Abacus Insights results: 77% accuracy at roughly half the cost for a platform-native agent, plus about 40% less manual effort and onboarding to first value in roughly half the time.


### [Using AI\_Functions in Your Data Warehouse: Top Use Cases](https://yomu.fyi/post/using-ai-functions-in-your-data-warehouse-top-use-cases.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Srikant Das, Ismail Makhlouf
- Published: Aug 14, 2026

Databricks AI Functions bring model inference into SQL queries, addressing the need to combine warehouse-resident structured data with unstructured reviews, tickets, PDFs, and transcripts without exporting rows to a separate service. The post presents six Lakehouse use cases: document parsing and extraction, sentiment classification, translation, ticket routing, sales-call extraction, and generative drafting with ai\_query. Task-specific functions such as ai\_parse\_document, ai\_extract, ai\_classify, and ai\_translate handle specialized work, while ai\_query sends prompts to an accessible Databricks-hosted Foundation Model serving endpoint. Unity Catalog permissions, existing pipelines, and system.billing.usage keep governance and billing within Databricks, while SQL execution manages planning, parallelization, and retries. Production guidance includes tagging jobs, sampling at least 10,000 rows, choosing models deliberately, requesting structured output, and versioning prompts.


### [How Scottish Water Made Its Capital Investment Data Conversational With Databricks Genie](https://yomu.fyi/post/how-scottish-water-made-its-capital-investment-data-conversational-wit.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sourav Gulati, Poppy Harvey
- Published: Aug 13, 2026

Scottish Water’s Capital Investment teams had project data, reports, and metrics available, but non-technical users often struggled to find answers, creating duplicated reporting and delays. SPARK addresses this access problem through a natural-language interface in Microsoft Teams: Copilot routes questions via a supervisor agent and MCP to the Databricks Genie Agent, which queries governed Unity Catalog data. The implementation curates use-case data in a gold layer, adds metric views as a semantic layer, and configures business rules, examples, benchmarking, feedback, and monitoring for accuracy and performance. Environment-parameterised Databricks Asset Bundles, separate environments, Azure DevOps approvals, Entra ID, and Key Vault support repeatable deployment. The post reports lookup journeys shrinking from around eight clicks or four to five search steps to a direct question, with an estimated 520 to 1,300 hours saved annually under a stated usage scenario.


### [What are AI Hallucinations?](https://yomu.fyi/post/what-are-ai-hallucinations.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 13, 2026

AI hallucinations are coherent, confident outputs that are factually wrong, fabricated, or unsupported, appearing in chatbots, image generators, and multimodal systems. Generative models predict likely continuations rather than retrieve guaranteed facts, while training incentives, limited or flawed data, generation settings, and error compounding in reasoning models can increase the risk. Examples include Google Bard’s incorrect JWST claim, Air Canada’s nonexistent bereavement policy, Microsoft Sydney’s erratic responses, and fabricated legal citations that led to sanctions. Hallucinations create enterprise risks involving safety, liability, compliance, security, reputation, and trust; recommended controls include guardrails, source verification, adversarial testing, monitoring, human review, retrieval-augmented generation, governed data, and evaluation throughout development and deployment.


### [Smart Routing in Unity AI Gateway: Match frontier quality with 30%+ lower cost per task](https://yomu.fyi/post/smart-routing-in-unity-ai-gateway-match-frontier-quality-with-30-lower.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ankit Mathur, Ivan Zhou, Bryan Qiu, Rohit Agrawal, Elise Gonzales, Kelly Albano
- Published: Aug 13, 2026

Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.


[Newer posts](https://yomu.fyi/company/databricks.md) · [Older posts](https://yomu.fyi/company/databricks/page/3.md)