---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [From months to minutes: Building real-time clinical data pipelines with natural language](https://yomu.fyi/post/from-months-to-minutes-building-real-time-clinical-data-pipelines-with.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matthew Giglia, Tim Kessler, Assunta Carey-Saylor
- Published: Apr 28, 2026
- Type: Explainer

Healthcare data teams often spend months integrating EHR systems, normalizing HL7, CCD, and X12, and routing data through intermediary storage before analytics, creating latency and maintenance burden. Databricks and Redox describe a pipeline model that combines Redox MCP Server, natural-language prompts, and Databricks Zerobus Ingest to build integrations inside Databricks and stream clinical data directly into Unity Catalog managed tables. The post says Zerobus provides subsecond latency, while the MCP Server identifies environments, suggests workflow steps, executes integration tasks, and surfaces validation signals such as logs and performance summaries. A demonstrated workflow retrieved a recent patient admission as structured data with a plain-language summary, and Redox writeback can return AI outputs to EHRs for point-of-care action. The same foundation is presented as enabling real-time use cases and Redox Agents built with Databricks Genie Spaces.


### [Agentic data engineering with Genie Code and Lakeflow](https://yomu.fyi/post/agentic-data-engineering-with-genie-code-and-lakeflow.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Gal Oshri, Camiel Steenstra, Lennart Kats, Joanna Zouhour
- Published: Apr 28, 2026
- Type: Explainer

Genie Code is presented as a natural-language assistant for developing, orchestrating, deploying, and debugging data pipelines and jobs. It uses pipeline and job context, including code, configuration, run results, Unity Catalog metadata, lineage, popularity, and code samples, to help engineers discover datasets and understand data flows. Engineers can describe pipelines or jobs, and Genie Code can generate Spark Declarative Pipelines with Bronze, Silver, and Gold layers, sources, transformations, data quality expectations, and outputs, then configure orchestration, schedules, dependencies, Auto Loader, AutoCDC flows, and Declarative Automation Bundles. It also analyzes failures and unexpected row-count or schema changes, proposes cross-file updates with reviewable diffs, and supports extensions through custom instructions, agent skills, and MCP servers. The stated result is faster development and guided debugging while workflows remain aligned with Unity Catalog governance, performance, and data quality standards; future plans include background failure response and cluster right-sizing.


### [Securely send first-party conversion signals with Snapchat Conversions API on Databricks Marketplace](https://yomu.fyi/post/securely-send-first-party-conversion-signals-with-snapchat-conversions.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nikhil Gaekwad, Jen McNamee, Katy Yuan
- Published: Apr 28, 2026
- Type: Announcement

Snapchat Conversions API is now available as a Databricks Marketplace listing that lets teams activate first-party conversion signals directly from governed Lakehouse data. The included pre-built notebook reads gold-layer tables, batches web, app, and offline events, and supports deduplication with Snap Pixel, allowing server-side signals without double-counting. Because it runs in the Databricks environment, Unity Catalog preserves governance, lineage, security, and access controls while teams control what data is shared and when. The post says fuller server-side coverage can raise Event Match Quality, while citing that advertisers who improved their EQS from “Poor” to “Good” saw 26% higher ROAS and 49% lower CPI. It also describes fresher custom audiences and Lookalikes plus more complete deduplicated measurement.


### [How leading tech companies are killing the builder's tax with Lakebase](https://yomu.fyi/post/how-leading-tech-companies-are-killing-the-builder-s-tax-with-lakebase.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amey Banarse, Madelyn Mullen
- Published: Apr 27, 2026
- Type: Explainer

The post argues that AI-native applications face a “builder’s tax” because operational databases and lakehouse analytics are separated by CDC, ETL/ELT, and reverse ETL pipelines. This architecture can leave AI systems with stale data while duplicating governance and shifting engineering effort toward orchestration and failure management. It presents Lakebase, a fully managed serverless Postgres engine integrated with the Databricks Data + AI Platform, as a shared operational foundation where applications, agents, analytics, and governance use the same data. The proposed pattern combines a lakehouse intelligence layer, a low-latency Lakebase execution layer, and a continuous learning loop, with pgvector supporting agent memory and semantic search. Reported customer examples include faster feature onboarding, reduced on-call disruption, higher agent throughput, and consolidated data estates, though the results vary by organization.


### [Powering KPMG UK Audit's AI future with Databricks](https://yomu.fyi/post/powering-kpmg-uk-audit-s-ai-future-with-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Mark Wallington, Greta Nasai
- Published: Apr 27, 2026
- Type: Problem & solution

KPMG UK is evolving its Audit data platform with Databricks to support AI-enabled analytics while preserving governance, rigour, and professional standards. The program converges structured data, advanced analytics, and AI on a unified cloud-native Lakehouse, with Databricks SQL replacing core SQL Server workloads and Delta underpinning the platform. Lakebridge assessed migration complexity, while Databricks-hosted large language models including Claude Sonnet and Genie Code helped convert T-SQL, refactor stored procedures, modularize queries, and suggest Delta-based optimizations under engineer review. This reduced refactoring time by around 60% and enabled modernization of more than 400 scripts and stored procedures in roughly three months. Databricks SQL Serverless provides elastic compute for spiky workloads, while Genie offers traceable, version-controlled SQL and Delta Sharing supports governed data exchange.


### [Inside one of the first production deployments of Lakebase: LangGuard's agentic workflow governance engine](https://yomu.fyi/post/inside-one-of-the-first-production-deployments-of-lakebase-langguard-s.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Venkat Raghavan, Jason Keirstead, Ravi Srinivasan, Nina Williams, Amelia Westberg
- Published: Apr 27, 2026
- Type: Explainer

LangGuard is presented as a runtime enforcement layer for enterprise agentic workflows, monitoring actions, decisions, tools, credentials, and intent across connected systems. Its GRAIL data fabric records multidimensional trace data, builds a live knowledge graph, and evaluates allow/deny/modify decisions against policy before tools, datasets, or models are accessed. The deployment uses Databricks Lakebase as the operational system of record, relying on PostgreSQL, serverless autoscaling, scale-to-zero, compute-storage disaggregation, hot-data caching, and copy-on-write branching for policy testing. LangGuard chose this architecture to handle bursty trace writes and low-latency enforcement reads without provisioning for peak demand, while keeping operational data available to Databricks analytics and AI capabilities without additional ETL. The stated next step is predictive governance: training behavioral models on historical traces to flag anomalous agent behavior before a policy violation.


### [The next generation of Databricks Genie](https://yomu.fyi/post/the-next-generation-of-databricks-genie.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ken Wong, Dillon Morrison, Richard Tomlinson
- Published: Apr 26, 2026
- Type: Announcement

The next generation of Databricks Genie expands the assistant beyond individual Genie Spaces, combining structured and unstructured enterprise data in a unified chat experience. It reuses logic from certified Genie Spaces, governed dashboards, and Databricks Apps, with metadata-based routing that prioritizes higher-trust sources, while new reasoning models and agent architecture handle questions spanning multiple domains. Built-in connectors for Google Drive and SharePoint, plus MCP support, let Genie access knowledge stores and take actions; Unity Catalog AI Gateway manages these connections. The experience replaces Databricks One as an account-level global home, adding domains, custom URLs, unified login, automated identity management, and governance through Unity Catalog. Native iOS and Android apps extend access to dashboards, apps, and chat beyond desktop environments.


### [Model risk management in 2026: A banker's guide to the revised interagency guidance](https://yomu.fyi/post/model-risk-management-in-2026-a-banker-s-guide-to-the-revised-interage.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pavithra Rao, Jennifer Miller, Chaitanya Varanasi, Kim Hatton
- Published: Apr 25, 2026
- Type: Problem & solution

The post presents the April 17, 2026 revision to model risk management guidance as a shift toward risk-based, principles-driven oversight, replacing several earlier issuances from the Federal Reserve, FDIC, and OCC. It identifies five operational changes: materiality tiering, end-to-end lifecycle governance, reproducible effective challenge, continuous drift and performance monitoring, and application of MRM principles to GenAI and agentic systems. Its proposed response is a Databricks reference architecture built on Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines, Feature Store, MLflow, Model Registry, Model Serving, and assurance tools. The design makes tiering metadata-driven and turns lineage, validation, approvals, monitoring, documentation, and retirement records into evidence generated during normal model work. The stated conclusion is that a unified substrate can reduce integration and manual evidence work, allowing regulatory changes to become configuration exercises rather than multi-quarter programs.


### [OpenAI GPT-5.5 + Codex, now available and fully governed on Databricks](https://yomu.fyi/post/openai-gpt-5-5-codex-now-available-and-fully-governed-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Patrick Wendell, Hanlin Tang, Ahmed Bilal, Arnav Singhvi, Ivan Zhou, Harish Gaur
- Published: Apr 24, 2026
- Type: Announcement

Databricks announces native support for OpenAI’s GPT-5.5, making it available for coding workflows with Codex, enterprise agents, document pipelines, and data-driven employee workflows. Access is governed through Unity AI Gateway, which provides permissions and rate limits, configurable guardrails for PII, prompt injection, and content safety, MCP tool-call auditing, failover, and request-level observability for model and Codex interactions. The announcement describes GPT-5.5 use with Genie for natural-language analytics, Agent Bricks Custom Agents for multi-step workflows, and Lakeflow Spark Declarative Pipelines for document-focused GenAI ETL involving summarization, extraction, and classification. GPT-5.5 is available on Databricks across AWS, Azure, and GCP, with entry points including AI Playground, governed Codex endpoints, Agent Bricks, and the Foundation Model API.


### [Operational databases: How they work and when to use them](https://yomu.fyi/post/operational-databases-how-they-work-and-when-to-use-them.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 24, 2026
- Type: Explainer

Operational databases, or online transaction processing (OLTP) databases, support day-to-day operations by processing real-time CRUD transactions with low latency, high concurrency, and ACID guarantees. The guide contrasts them with data warehouses and OLAP systems, which favor historical analysis, complex aggregations, and read-heavy workloads, while describing ETL, CDC, and operational data stores in the OLTP-to-OLAP pipeline. It argues that traditional OLTP systems can struggle with real-time analytics and AI because of siloed data, structured-data constraints, rigid schemas, scaling limits, and missing governance features. Databricks Lakebase is presented as a hybrid solution using separate storage and compute, serverless Postgres, elastic scaling, branching, shared lakehouse storage, and open formats. The proposed workflow connects existing OLTP systems through CDC or streaming into Delta Lake for fresh SQL, BI, ML, and AI access.


### [Databricks partners with OpenAI on GPT-5.5](https://yomu.fyi/post/databricks-partners-with-openai-on-gpt-5-5.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Hanlin Tang, Ahmed Bilal, Arnav Singhvi, Ivan Zhou, Harish Gaur
- Published: Apr 23, 2026
- Type: Announcement

Databricks announces a partnership with OpenAI around GPT-5.5, described as OpenAI’s strongest frontier model for enterprise agentic work, complex document reasoning, and long-horizon coding agents. The model powers Codex and is presented as able to research, analyze data, create documents and spreadsheets, operate software, use tools, check outputs, recover from ambiguity, and continue through multi-part tasks. Databricks evaluated it on OfficeQA, a benchmark built from 89,000 pages of U.S. Treasury Bulletins that tests document retrieval, table interpretation, and precise calculation. With retrieval handled, GPT-5.5 scored 64.66% versus GPT-5.4’s 57.14%; in the full-agent OfficeQA Pro Agent Harness, it scored 52.63% versus 36.10%, representing reported improvements of roughly 13% and a 46% reduction in errors.


### [Guide to BI Reporting and Maximizing Intelligence Effectiveness](https://yomu.fyi/post/guide-to-bi-reporting-and-maximizing-intelligence-effectiveness.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 23, 2026
- Type: Explainer

Business intelligence (BI) reporting turns data from operational systems and analytical repositories into structured reports, dashboards, and visualizations for organizational decision-making. The guide explains managed and ad hoc reporting, then traces a typical workflow through data sources, ETL, deduplication, normalization, validation, data modeling, visualization, and automated distribution. It compares established platforms such as Microsoft Power BI, Tableau, SAP BusinessObjects, Qlik, and Looker with newer AI-native tools, while noting evaluation criteria including connectivity, scale, governance, usability, and cost. It also addresses dashboard overload, stale semantic models, and specialist bottlenecks, and cites reports of 10x faster query creation from AI-assisted natural-language querying alongside recommendations to start with one decision, one primary source, and three to five KPIs.


### [Announcing the Public Preview of Lakeflow Designer](https://yomu.fyi/post/announcing-the-public-preview-of-lakeflow-designer.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jason Messer, Emanuel Zgraggen, V Maharajh, Matt Jones, Tracy Yang
- Published: Apr 23, 2026
- Type: Announcement

Databricks announces the Public Preview of Lakeflow Designer, a visual, no-code, AI-native experience for data preparation and analytics built directly into the Databricks platform. It targets analysts, domain experts, and other less technical users with drag-and-drop workflows and natural-language authoring, while representing each transformation as an operator with previews. Genie Code can use Unity Catalog metadata, descriptions, lineage, popularity, and example queries to generate or modify workflows, inspect transformation results, and iterate when needed. Every visual transformation produces production-ready Python code that can be reviewed, versioned in Git, and scheduled through Lakeflow Jobs, while users pay for compute rather than per-user licenses. The preview is available in all workspaces, although an administrator may need to enable Visual data prep in the preview portal.


### [Are LLM agents good at join order optimization?](https://yomu.fyi/post/are-llm-agents-good-at-join-order-optimization.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Eric Liang, Ryan Marcus, Sid Taneja, Yuhao Zhang
- Published: Apr 22, 2026
- Type: Benchmark

Join ordering is difficult because relational optimizers choose among exponentially many plans and depend on potentially inaccurate cardinality estimates. Databricks and UPenn researchers built an LLM agent that uses structured outputs to generate valid join reorderings and executes candidates through a tool returning runtime and intermediate subplan sizes. It can test up to 50 iterations, while the evaluation tested 15 rollouts per query across 113 Join Order Benchmark queries after duplicating each dataset row ten times. With a frontier model, the best tested orders improved geometric-mean latency by 1.288 times and reduced P90 latency by 41% versus the standard Databricks optimizer. A five-way example links its gain to LIKE predicates that are exceptionally difficult for cardinality estimators.


### [How conversational analytics removes the BI bottleneck](https://yomu.fyi/post/how-conversational-analytics-removes-the-bi-bottleneck.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Catherine Brown
- Published: Apr 22, 2026
- Type: Explainer

The interview presents conversational analytics as a way to move beyond fixed dashboards and turn governed data into decisions and workflows. Databricks’ Genie lets nontechnical users ask open-ended questions in plain language across Lakebase, Lakehouse, and federated sources, while Unity Catalog applies shared definitions and access controls. Lakebase provides transactional storage, instant provisioning, environment forking without copying underlying data, and scale described as billions or trillions of records; examples include real-time matchmaking, routing, and purchase decisions. The discussion stresses that human oversight, refreshed data, business context, and controls against hallucination remain necessary, citing Fox Sports’ chatbot deployment as an example. It concludes that organizations combining trusted semantics with operational data systems can make faster decisions and automate work, while the competitive gap may compound for slower adopters.


### [How to transform document activation workflows with Genie and Agent Bricks](https://yomu.fyi/post/how-to-transform-document-activation-workflows-with-genie-and-agent-br.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser
- Published: Apr 22, 2026
- Type: Explainer

Enterprise document workflows often require people to read PDFs, copy fields into spreadsheets, and re-enter data into business systems, creating delays, error risk, and weak auditability. The proposed alternative combines AI/BI Genie for natural-language queries over Delta tables, Agent Bricks for production agents, and Unity Catalog for permissions, lineage, and traceability. A five-phase workflow extracts fields into Lakeflow Bronze, Silver, and Gold layers, then supports structured queries, clause-level RAG answers, multi-agent routing, and actions through MCP servers. Examples include syncing validated rights data to ERP or CRM systems, updating billing entitlements, and triggering workflows from detected obligations. The approach targets media, advertising, ad tech, and telecommunications, where customers see improvements including faster month-end close, recovered revenue, reduced leakage, and lower operational risk.


### [Beyond the spreadsheet: How Databricks is delivering the modern CFO in financial services](https://yomu.fyi/post/beyond-the-spreadsheet-how-databricks-is-delivering-the-modern-cfo-in.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jennifer Miller, Marcela Granados, Andrea DeSosa, Alex Oberlander, Kim Hatton, Pavithra Rao, Naeem Rehman, Pravin Varma, Olga Deriy, Prasanna Selvaraj
- Published: Apr 22, 2026
- Type: Explainer

Financial-services CFOs are being asked to act as Strategists and Catalysts, but fragmented legacy systems, T+1 batch processing, opaque reporting logic, and mismatched business semantics keep them focused on stewardship and retrospective operations. The proposed answer is Databricks as a unified, governed platform combining real-time streaming, centralized lineage, data and AI, with Unity Catalog, Lakeflow, Genie, and Agent Bricks addressing trust, latency, access, and model reproducibility. Unity Catalog can connect semantic definitions and end-to-end lineage from transactions through regulatory reports and models, while Lakeflow supports continuous ledger and liquidity processing. The post cites a global bank reducing liquidity-reporting processing from 10 hours to 8 minutes and Nationwide Insurance reporting a 5-point combined-ratio improvement and 3-point expense-ratio improvement. It presents a Lakehouse-based CFO stack as a shift from reporting historical results toward real-time capital management and previews AI-driven deposit and PPNR modeling.


### [AI App Development: Guide To Building AI-Powered Apps](https://yomu.fyi/post/ai-app-development-guide-to-building-ai-powered-apps.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 22, 2026
- Type: Tutorial

Production-grade AI app development combines model orchestration, transactional state, governance, security, and live-data integration rather than only interface construction. The guide recommends defining users, outcomes, success metrics, timelines, and AI-relevant journeys, then shipping must-have features before enhancements. It evaluates builders by end-to-end support, technical audience, platform integration, code export, CI/CD, data connections, pricing, and deployment limits. For Databricks teams, it describes Databricks Apps, Declarative Automation Bundles, and Lakebase: serverless app compute, version-controlled deployment, and managed PostgreSQL with synced Unity Catalog Delta tables. It also recommends monitoring outputs, collecting correction feedback, refining prompts with automated evaluations, and governing models through MLflow, concluding that integrated infrastructure can leave teams focused on production AI workflows.


### [IoT in Manufacturing: Strategy, Components, Use Cases, and Challenges](https://yomu.fyi/post/iot-in-manufacturing-strategy-components-use-cases-and-challenges.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 22, 2026
- Type: Explainer

IoT in manufacturing is presented as an operational architecture for collecting machine data, improving production and supply-chain visibility, and preventing equipment failures under strict latency, reliability, and safety requirements. The guide covers device selection, sensor placement, tiered sampling, edge-to-cloud pipelines, platform criteria, security, compliance, workforce training, and a phased deployment roadmap. Its technical split sends high-frequency vibration and acoustic signals to edge gateways for local processing while cloud systems support stateful analytics, predictive maintenance scoring, and model training; protocols such as MQTT, OPC-UA, AMQP, and Modbus are evaluation criteria. Reported examples include multi-factory OEE monitoring across more than 200 production lines, reducing reporting lag from 24 hours to under five minutes, digital twins for maintenance simulation, and IoT logistics routing. The action plan recommends starting with predictive maintenance and OEE monitoring on one line before scaling.


### [Multimodal Data Integration: Production Architectures for Healthcare AI](https://yomu.fyi/post/multimodal-data-integration-production-architectures-for-healthcare-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Maks Khomutskyi
- Published: Apr 22, 2026
- Type: Problem & solution

Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.


[Newer posts](https://yomu.fyi/company/databricks/page/26.md) · [Older posts](https://yomu.fyi/company/databricks/page/28.md)