Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
429 postsWebsite
Latest articles
Data Mesh vs. Data Fabric: Key Differences and How the Lakehouse Resolves the Debate
Data mesh and data fabric address different constraints in distributed data architecture: mesh changes ownership and accountability, while fabric automates integration and governance. A data fabric uses active metadata, machine learning, centralized policy enforcement, and virtualized access to discover, govern, and connect data across heterogeneous environments with less data movement. A data mesh assigns domain teams responsibility for data products, quality, reliability, and lifecycle management, supported by self-serve infrastructure and federated computational governance. The article presents a lakehouse as a substrate where domains publish products while capabilities such as Unity Catalog, Delta Sharing, and Lakehouse Federation provide metadata, governance, and sharing. Its conclusion is that the approaches are complementary: organizations should identify whether organizational bottlenecks or technical fragmentation is the dominant constraint, then consider a hybrid.
Databricks StaffModernizing SQL ETL in Lakehouse with Declarative Patterns
Databricks is extending declarative ETL patterns into the Lakehouse SQL Editor so SQL practitioners can define recurring transformations without a dedicated pipelines-oriented environment. The initial operations cover append-only updates, change data capture through AUTO CDC, and targeted batch overwrites through REPLACE WHERE; flows can be scheduled, triggered by upstream updates, run on demand, or orchestrated through SQL tasks in Jobs. APPEND tracks processed source data, AUTO CDC handles keys, sequencing, deletes, and SCD Type 1 or Type 2 results, while Enzyme can incrementally process changed data for REPLACE WHERE. In Lakehouse benchmark testing, Enzyme-powered REPLACE WHERE ran 3.4x faster and 2.5x cheaper than traditional REPLACE WHERE, while users can retain procedural SQL and later move to the Lakeflow Pipelines Editor for multi-file development, dependencies, validation, and deployment.
Matt Jones, Shanelle RomanOpen Table Formats Explained: Iceberg vs. Delta vs. Hudi
Open table formats add metadata layers over Parquet or ORC files in object storage, enabling ACID transactions, schema evolution, time travel, and database-like consistency for data lakes. Apache Iceberg uses manifests and manifest lists in a metadata tree, supports partition evolution without rewriting existing files, and creates immutable snapshots for auditing, rollback, and reproducible queries. Delta records atomic writes in an ordered transaction log with checkpoints and is closely associated with Spark, while Hudi uses record-level indexes for frequent updates, change-data-capture, and near-real-time ingestion. All three share core capabilities but differ in partition evolution, update and delete strategies, and multi-engine support; Parquet and ORC remain file formats rather than table formats. The overview recommends scheduled compaction, metadata maintenance, health monitoring, and a production-volume proof of concept when selecting a format.
Databricks StaffRun, debug, and scale Databricks workloads from your local IDE
Databricks introduces updated IDE and CLI integrations that connect VS Code, Cursor, or a terminal directly to Databricks compute, workspace, and Unity Catalog. Existing Databricks Connect supported local Spark development, but remote non-Spark workloads and dependency synchronization remained pain points. The new SSH tunnel supports interactive execution and debugging of Python and SQL workloads on Serverless, AI Runtime, and dedicated clusters, while keeping files and dependencies aligned with Databricks Runtime and the workspace. CLI flags can select serverless, GPU-backed AI Runtime, or a dedicated cluster, and can start the tunnel in VS Code or Cursor; additional options configure base environments and usage policies, while the IDE can browse Unity Catalog assets.
Tanishq Maheshwari, Matt JonesHow Databricks Uses AI to Accelerate Incident Investigation
Databricks describes AI SRE, an AI-powered debugging agent for operating hundreds of microservices across more than 1,500 Kubernetes clusters, 70+ regions, and three clouds. It addresses incident investigations by assembling context from platform health, service telemetry, changes, dependencies, and team-specific runbooks, with automatic triage followed by interactive, natural-language investigation. The platform uses layered primitives, purpose-built APIs, and a core orchestration engine; team-owned agentic runbooks encode domain procedures, while controlled access and evidence links support auditing. The post reports support for more than 150 teams, 250+ weekly active users, and over 2,000 daily investigations, with users saving several hours of debugging time.
Avijeet Gupta, Bhuban Seth, Kusum MadarasuRelational vs Non-Relational Database: Choosing the Right Data Store
Selecting between relational and non-relational databases requires matching the data store to requirements for structure, consistency, scale, and query complexity. Relational systems organize data in predefined tables, enforce schemas and keys, and provide ACID transactions, making them suited to structured, transactional workloads and complex joins. Non-relational systems support documents, key-value pairs, graphs, or wide-column structures, often using flexible schemas and schema-on-read while scaling horizontally and accepting eventual consistency for throughput and availability. The trade-offs include vertical versus horizontal scaling, SQL versus database-specific query languages, and database-enforced versus application-level integrity. The guidance recommends documenting workload needs, prototyping with realistic volumes and failure scenarios, and considering polyglot persistence when different workloads require specialized systems.
Databricks StaffTransactional Vs Analytical Database: Choosing OLTP, OLAP, or Hybrid
Transactional and analytical databases serve different workloads: OLTP handles high-volume, real-time reads and writes with ACID guarantees, while OLAP supports complex queries over large historical datasets for reporting and business intelligence. The comparison focuses on trade-offs involving latency, concurrency, data freshness, schema design, transaction size, and storage format. Row-oriented storage and normalized schemas support precise operational updates, whereas column-oriented storage and denormalized designs improve compression, scans, and aggregations across millions of rows. The source describes transactional systems as foundational for applications such as banking, e-commerce, healthcare, and reservations, while analytical systems support dashboards, forecasting, and ad-hoc analysis. Its recommended architecture is to run both systems independently and connect them with Change Data Capture, balancing operational reliability with analytical throughput.
Databricks StaffConnecting retail demand planning to campaign and store execution
Retail and consumer goods teams often split performance review, demand planning, marketing activation, store execution, and measurement across disconnected systems, making it difficult to connect analysis with results. The Databricks demo presents a unified application built on data unification, governance, and intelligence, combining point-of-sale, loyalty, supply chain, media spend, inventory, and store-operations signals with role-specific access. Users can move from KPI and forecast-versus-actual analysis to SKU or store what-if planning, audience selection, campaign activation, replenishment, price-tag updates, assigned tasks, and post-campaign measurement. Genie supports natural-language querying, while specialized agents and a coordinating supervisor connect sales insights, demand planning, audience building, in-store operations, and measurement. The stated design goal is a governed, shared workflow that lets teams test decisions, execute them across stores, and evaluate outcomes without losing operational context.
Jack Yallop, Pavi SinghInbound Private Link now supports account-level Genie One, the account console, and custom URLs
Databricks has expanded Inbound Private Link in Beta on AWS and Azure to cover account-level resources, including account-level Genie One, the account console, Governance Hub, and account-level APIs. The update also supports custom URLs and Managed Disaster Recovery stable URLs, while one shared General Access endpoint in any region can serve workspace and account-level UI and API resources, removing the need for one endpoint per region or workspace. These capabilities use context-based ingress, where account admins define allow and deny rules by caller identity, network source, and destination through account and workspace policies. Enabling private access to account-level resources requires registering and allowlisting a General Access endpoint with the account-policy and resolving the custom URL to that endpoint. Existing workspace-specific URLs, private access settings, and IP access lists continue to work in parallel, while service-direct and SCC relay endpoints retain per-region requirements.
Robert Zhang, Manish Bansal, Chen He, Yankai ZhangBusting SQL Migration Myths: How New SQL Features Make Lift-and-Shift to Lakehouse Easier
The post addresses the procedural core of warehouse-to-lakehouse migrations: legacy stored procedures that use temporary tables, cursors, control flow, multi-table updates, and transaction handling while remaining maintained by SQL teams. Using a composite daily-orders procedure based on an Oracle migration, it translates rather than rewrites the logic on Databricks, covering procedure registration in Unity Catalog, session-scoped temporary tables, cursor syntax, handlers, conditional scripting, and atomic transactions. Databricks SQL scripting supports cursors from Runtime 18.1, and BEGIN ATOMIC provides automatic commit or rollback plus row-level conflict detection; catalogManaged tables are required inside atomic blocks. The post says this mechanical translation can cut migration timelines by 50–75% for complex procedures, while preserving business logic and allowing the SQL team to maintain it, though the procedure still needs validation in a migration project.
Abhishek Dey, Laurent LéturgezDesigning effective Genie Agents from a single prompt
Generic agents may select the first relevant-looking revenue table rather than the finance-maintained source because they lack business context. Genie Agents address this by using a single prompt to assemble a domain-specific agent from governed Unity Catalog context, including structured data, documents, and files. The prompt identifies the desired outcome and relevant sources, while the agent can retrieve content from PDFs, Word documents, presentations, images, and tables under the requesting user’s permissions. The recommended process is to begin with one focused, repetitive job, test it against known questions or incidents, and expand only after measuring performance. Built-in benchmarks provide accuracy scores, while monitoring surfaces user questions and feedback that can reveal gaps in the curated context.
Megan TupperDatabricks Document Intelligence: pushing the frontier for complex document extraction
Databricks introduces Precision Mode for its ai_extract document extraction API, targeting enterprise workloads that challenge existing LLM and rules-based systems. The mode addresses long documents with cross-references, large nested outputs such as thousands of line items, and schemas requiring reasoning across financial statements or applying discounts. It combines custom-trained, efficient extraction models with an agent harness that decomposes jobs semantically, runs smaller tasks in parallel, preserves intermediate results, and reconciles them into a final structured output. On benchmarks covering roughly 9,000 documents, including files up to 2,000 pages and schemas with more than 300 deeply nested fields, Precision Mode reached 94.7% accuracy, seven points above the strongest frontier-model chunk-and-merge baseline, GPT-5.6 Sol; the source says it is now available through ai_extract and the Information Extraction UI.
Jane Zhang, Arnav Singhvi, Ivan Zhou, Archika Dogra, Matthew Ding, Nihit DesaiWhen it comes to Governance, Retailers need a control plane for context
Retailers are moving from isolated generative-AI pilots to enterprise-wide adoption, where each store, headquarters, and digital workflow needs different data, permissions, models, and cost controls. The post argues that useful AI depends on governed business context, while fragmented tools can produce separate provider contracts, logs, permission models, cost centers, and answers to the same question. It proposes an open control plane that centralizes governance and model access without forcing employees into one application or vendor. The described Databricks design combines Unity Catalog for data and business context, Unity AI Gateway for routing and oversight, Foundation Model APIs for commercial, open-source, and custom models, and Genie for natural-language queries over governed data; the intended result is faster experimentation and deployment with visibility into access, spending, lineage, and auditability.
BJ Sullivan, Sarah Duffy, Tayo Olabumuyi, Logan BoydEvaluating AI Agents Live at the Grounded Reasoning Cup
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
Databricks AI Research TeamHow Databricks Feature Store serves features with sub-second freshness
Databricks Feature Store addresses the lag between batch-computed machine-learning features and decisions that depend on events from seconds earlier. It lets a feature definition drive both offline batch processing and online streaming pipelines, orchestrating Kafka ingestion, Spark Real-Time Mode (RTM), Lakebase, and Model Serving. For a rolling 10-minute transaction sum, RTM updates per entity state in local RocksDB, expires contributions per event, and writes revised values to Lakebase through a streaming JDBC sink. The described path reports end-to-end p99 200ms latency from Kafka arrival to online feature availability, while Model Serving retrieves declared dependencies automatically at inference. Tumbling and sliding windows remain options when fewer, cheaper updates are acceptable.
Ian Ackerman, Nick Joung, Abhay BothraThe prototyping tax is killing your AI roadmap
The post argues that the “prototyping tax”—delay between an AI-powered idea and a working MVP—causes initiatives to lose momentum, often because organizational context is fragmented across teams, tools, and codebases. Traditional encapsulation and siloed domain knowledge can leave agents with syntax and contracts but not business intent, making production codebases harder to navigate than personal projects. The proposed response is agentic development grounded in business semantics and governed data, with alignment occurring through working builds while CI/CD, code review, and production rigor remain unchanged. The post recommends tracking time-to-prototype, first-pass acceptance rate, and PoC-to-production rate to distinguish faster demos from durable delivery. It cites a benchmark of 401 data tasks and Abacus Insights results: 77% accuracy at roughly half the cost for a platform-native agent, plus about 40% less manual effort and onboarding to first value in roughly half the time.
Navdeep Alam, Zhe SunUsing AI_Functions in Your Data Warehouse: Top Use Cases
Databricks AI Functions bring model inference into SQL queries, addressing the need to combine warehouse-resident structured data with unstructured reviews, tickets, PDFs, and transcripts without exporting rows to a separate service. The post presents six Lakehouse use cases: document parsing and extraction, sentiment classification, translation, ticket routing, sales-call extraction, and generative drafting with ai_query. Task-specific functions such as ai_parse_document, ai_extract, ai_classify, and ai_translate handle specialized work, while ai_query sends prompts to an accessible Databricks-hosted Foundation Model serving endpoint. Unity Catalog permissions, existing pipelines, and system.billing.usage keep governance and billing within Databricks, while SQL execution manages planning, parallelization, and retries. Production guidance includes tagging jobs, sampling at least 10,000 rows, choosing models deliberately, requesting structured output, and versioning prompts.
Srikant Das, Ismail MakhloufHow Scottish Water Made Its Capital Investment Data Conversational With Databricks Genie
Scottish Water’s Capital Investment teams had project data, reports, and metrics available, but non-technical users often struggled to find answers, creating duplicated reporting and delays. SPARK addresses this access problem through a natural-language interface in Microsoft Teams: Copilot routes questions via a supervisor agent and MCP to the Databricks Genie Agent, which queries governed Unity Catalog data. The implementation curates use-case data in a gold layer, adds metric views as a semantic layer, and configures business rules, examples, benchmarking, feedback, and monitoring for accuracy and performance. Environment-parameterised Databricks Asset Bundles, separate environments, Azure DevOps approvals, Entra ID, and Key Vault support repeatable deployment. The post reports lookup journeys shrinking from around eight clicks or four to five search steps to a direct question, with an estimated 520 to 1,300 hours saved annually under a stated usage scenario.
Sourav Gulati, Poppy HarveyWhat are AI Hallucinations?
AI hallucinations are coherent, confident outputs that are factually wrong, fabricated, or unsupported, appearing in chatbots, image generators, and multimodal systems. Generative models predict likely continuations rather than retrieve guaranteed facts, while training incentives, limited or flawed data, generation settings, and error compounding in reasoning models can increase the risk. Examples include Google Bard’s incorrect JWST claim, Air Canada’s nonexistent bereavement policy, Microsoft Sydney’s erratic responses, and fabricated legal citations that led to sanctions. Hallucinations create enterprise risks involving safety, liability, compliance, security, reputation, and trust; recommended controls include guardrails, source verification, adversarial testing, monitoring, human review, retrieval-augmented generation, governed data, and evaluation throughout development and deployment.
Databricks StaffSmart Routing in Unity AI Gateway: Match frontier quality with 30%+ lower cost per task
Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.
Ankit Mathur, Ivan Zhou, Bryan Qiu, Rohit Agrawal, Elise Gonzales, Kelly Albano