---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Enabling Evolutionary Database Development: database branching with Lakebase](https://yomu.fyi/post/enabling-evolutionary-database-development-database-branching-with-lak-lakebase.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pramod Sadalage, Kevin Hartman
- Published: May 29, 2026

Evolutionary database development treats migrations as first-class CI/CD artifacts, but shared development databases still limit per-developer isolation and fast, realistic feedback. The post follows Jen as she adds location, batch, and serial fields to inventory, coordinating application changes, production-data preservation, schema migration, and tests for storing, reading, and displaying the new values. It contrasts shared databases and simplified local substitutes with Databricks Lakebase copy-on-write branches, created in one second with zero storage at creation, including for a terabyte-scale production database. Jen pairs code and database branches, while CI creates a temporary branch, validates an idempotent and reversible migration, runs application and database tests, and posts a schema diff, enabling combined review and shifting DBA attention toward design and maintainability.


### [AI Doesn't Scale Until You Stop Calling It Innovation](https://yomu.fyi/post/ai-doesn-t-scale-until-you-stop-calling-it-innovation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: May 29, 2026

Enterprises often stall between AI proofs of concept and production, and Philippe Rambach argues the remedy is to manage AI as a product rather than innovation. At Schneider Electric, Rambach built a 400-person AI organization split evenly between customer-facing products and internal operations, with business cases owned by lines of business and cross-functional scrum teams responsible through production and support. It standardizes a single core technology set, with Databricks managing infrastructure, data, and data flows, while gate reviews and quarterly portfolio decisions test technical readiness, commercial viability, and the business plan. Models are combined with context, guardrails, interfaces, forecasting, optimization, and real-time decisions; Microgrid Advisor reports up to a 20 percent reduction in energy costs, while Genie’s internal rollout remains early and accuracy is still being addressed.


### [Databricks at SIGMOD 2026](https://yomu.fyi/post/databricks-at-sigmod-2026.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Indrajit Roy
- Published: May 29, 2026

Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current\_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution.


### [Winning under CMS TEAM: Building the learning health system to realize success in VBC today and tomorrow](https://yomu.fyi/post/winning-under-cms-team-building-the-learning-health-system-to-realize.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kayla Grieme, Benjamin Goldsteen, Shankara Ettigi
- Published: May 29, 2026

CMS’s Transforming Episode Accountability Model (TEAM), beginning January 1, 2026, requires selected organizations to manage total cost and quality across five surgical episode categories and a 30-day post-discharge window. The post argues that retrospective dashboards and delayed claims data cannot support intervention or prepare hospitals for downside risk, especially amid complex integration across EHR, claims, post-acute, social-determinants, and provider data. It proposes a modern, cloud-native lakehouse foundation with unified data, operationalized AI/ML, embedded clinical workflows, and scalable architecture. Recommended capabilities include risk stratification, next-best-action decision support, post-acute optimization, provider benchmarking, and predictive performance alerts. In a composite 500-bed medical-center example, the approach is associated with 15% lower SNF costs, 8% better risk-capture completeness, and 12% fewer readmissions.


### [How enterprise leaders are scaling AI agents across their organization](https://yomu.fyi/post/how-enterprise-leaders-are-scaling-ai-agents-across-their-organization.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Christy Maver, Aly McGue
- Published: May 28, 2026

Executives from Danone, Capital One, Warner Bros. Discovery, Ford Credit, Gilead Sciences, and Databricks discuss scaling AI agents for measurable business value while maintaining governance, trust, and cost control. They describe deployment as only the first step, followed by monitoring, observability, performance assessment, and continuous learning, with risk reviews and ongoing evaluation integrated into the agent lifecycle. Organizations are shifting from single-task automation toward specialized multi-agent orchestration for complex workflows, while shadow capabilities and controlled sandboxes test accuracy without exposing live operations. Leaders recommend building momentum through low-risk wins such as Capital One’s Chat Concierge, training employees to prompt, and providing natural-language interfaces, certified data, consistent guardrails, and a secure architecture for deployment and monitoring.


### [Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance](https://yomu.fyi/post/advancing-apache-iceberg-on-databricks-iceberg-v3-ga-open-sharing-and.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jason Reid, Ryan Blue, Daniel Weeks, Michelle Leon
- Published: May 28, 2026

Databricks announces a broad set of Apache Iceberg capabilities in Unity Catalog, spanning General Availability, previews, and beta releases. Managed Iceberg is GA, supporting table creation, reads, writes, optimization, governance, and sharing, while Iceberg v3 adds deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables. Unity Catalog also federates external catalogs, vends credentials, shares live data with Iceberg REST-compatible clients through Delta Sharing, and applies attribute-based access control during server-side scan planning for supported external engines. These capabilities are presented as a unified approach to open APIs, cross-engine governance, zero-copy sharing, and production performance without copying data. The post also outlines Iceberg v4 and a proposal for Delta 5.0 to adopt an adaptive metadata tree structure.


### [Reliable LLM Inference at Scale](https://yomu.fyi/post/reliable-llm-inference-at-scale.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ying Chen, Wendy Hu, Ankit Mathur, Mike Eastham, Pei-Lun Liao, Wai Wu, Arjun DCunha
- Published: May 27, 2026

Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP\_NUM\_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.


### [BI Serving Pointers; Maximizing for Performance and TCO](https://yomu.fyi/post/bi-serving-pointers-maximizing-for-performance-and-tco.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Chris Koester
- Published: May 27, 2026

BI dashboards can become slow and expensive when teams respond to latency with separate aggregate tables, refresh pipelines, extracts, and tool-specific semantic layers. The post presents Databricks’ BI serving stack from physical storage through Unity Catalog’s governed semantic layer, recommending Gold-layer star schemas, managed tables, liquid clustering, and Predictive Optimization to reduce scanned data and improve query plans. Metric Views centralize KPI definitions and semantic metadata for dashboards, Genie, SQL notebooks, third-party BI tools, and AI agents, while materialization automatically maintains incremental pre-aggregations and routes queries transparently. Additional TCO guidance covers serverless SQL warehouse autoscaling, DBSQL disk and query-result caching, direct lakehouse connections, and system-table monitoring. The stated outcome is compounded lower latency and compute cost, including an observed average 22% performance improvement from Predictive Optimization and sub-second performance from materialized metrics.


### [Introducing Always-On pricing: automatic savings for Databricks Lakebase](https://yomu.fyi/post/introducing-always-on-pricing-automatic-savings-for-databricks-lakebas.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kunal Kande, Mike Jerome
- Published: May 27, 2026

Databricks introduces Always-On pricing for Lakebase, offering a lower rate for baseline compute while retaining elastic autoscaling. On any Lakebase Postgres Autoscaling project, operators turn off scale-to-zero and set an autoscaling range; the minimum capacity becomes the baseline, which receives the lower rate after 24 hours of continuous use. Capacity above the minimum continues to autoscale up to the configured maximum and is billed at standard Autoscaling rates, while users can re-enable scale-to-zero later. The pricing targets established workloads with a consistent activity floor; intermittent or new workloads can retain scale-to-zero to avoid paying for idle hours and uncertain baseline estimates. The announcement says the baseline price is 25% lower, and an additional 50% promotional discount runs through January 31, 2027.


### [How the lakebase architecture stays resilient to cloud failures](https://yomu.fyi/post/how-the-lakebase-architecture-stays-resilient-to-cloud-failures.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jasraj Dange, Hans Norheim
- Published: May 27, 2026

Lakebase is adapting its reliability architecture to agentic and on-demand workloads, which create far more control-plane activity, database starts, and pressure on finite cloud capacity. Lakebase uses stateless Postgres compute over zone-resilient storage, dedicated multi-availability-zone computes for optional high availability, and a vertically autoscaling virtualization layer that reduces dependence on cloud provisioning control planes. A dedicated data-plane controller is being developed for hot-path start and suspend operations, while cell-based regional deployments isolate failures and allow capacity to expand incrementally. The reliability program also uses component and whole-availability-zone failure injection, consistency checks, Service Level Indicators, and Service Level Objectives, including startup, failover, storage, and API measurements. The stated target is for every database to exceed 99.99% monthly availability; reported 2026 attainment through April shows 99.75% to 99.85% of databases meeting that threshold.


### [Announcing Lakebase Change Data Feed (CDF)](https://yomu.fyi/post/announcing-lakebase-change-data-feed-cdf.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pranav Aurora, Cheng Chen, Hristo Stoyanov
- Published: May 27, 2026

Lakebase Change Data Feed (CDF) is available in Public Preview to reduce the manual effort of moving data from operational databases into downstream systems. The feed is enabled once for all tables in a project, stored and governed in Unity Catalog Managed Tables, and readable by engines, models, and agents without separate extraction pipelines. From one shared feed, teams can build streaming pipelines with SDP, create materialized views with DBSQL, or compute and store embeddings with Agent Bricks, while consumers remain isolated from the primary operational workload. The announcement positions Lakebase as the native Bronze layer in a medallion architecture, complementing Synced Tables and providing governance and lineage across the data lifecycle.


### [Building a FHIR-native health data platform on Databricks Lakebase](https://yomu.fyi/post/building-a-fhir-native-health-data-platform-on-databricks-lakebase.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Marcin Jimenez, Aleksandr Kislitsyn, Nikolai Ryzhikov
- Published: May 27, 2026

Healthcare organizations often keep a FHIR server, analytics warehouse, and ETL pipelines separate, replicating clinical data and fragmenting governance. The proposed architecture standardizes incoming HL7v2, C-CDA, X12, and proprietary data into FHIR using Health Samurai converters, terminology normalization, MDM/MPI deduplication, and Implementation Guide validation. Aidbox, Health Samurai's FHIR server and database, runs on Databricks Lakebase, while Moonlink synchronizes operational and analytical formats without ETL. This exposes one governed dataset through Spark, SQL, ML, AI/BI, FHIR API, SMART on FHIR, and SQL on FHIR ViewDefinitions. The architecture supports use cases including EHR optimization, value-based care, member engagement, and compliance capabilities, with insights connected to clinician and billing workflows through SMART on FHIR and CDS Hooks.


### [AI readiness in telecommunications](https://yomu.fyi/post/ai-readiness-in-telecommunications.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Stephen Hage, Keerthi Josyula, Michael Zhang
- Published: May 26, 2026

Telecommunications companies are adopting AI for customer experience, network operations, and cost reduction, yet initiatives often stall before production because fragmented, ungoverned, semantically opaque data creates data debt. The post argues that AI readiness depends on a semantic layer unifying datasets and business definitions, governance, and catalog metadata across systems such as Oracle, Snowflake, Salesforce, ServiceNow, and Databricks. It presents Unity Catalog as the proposed foundation, using Delta Sharing, Lakeflow Connectors, and Lakehouse Federation to exchange, ingest, or query data without uniformly replicating it, while privilege-aware metadata and audit logging support compliance. Metric Views, lineage, tags, and glossaries give agents authoritative meanings for measures and terms such as revenue, ARPU, active user, and FTTH. The conclusion is that trustworthy operational AI requires a governed, unified data foundation and organizational commitment, not simply more capable models.


### [Pharma launch analytics: How to compress the first 90 days and win the three years that follow](https://yomu.fyi/post/pharma-launch-analytics-how-to-compress-the-first-90-days-and-win-the.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Adam Crown
- Published: May 23, 2026

Pharmaceutical launch analytics depends on compressing the time between data signals and commercial decisions, because early choices shape a trajectory measured over 12-to-36 months. The source frames the first 90 days as three phases: weeks 1–4 validate feeds, set NBRx and patient-start benchmarks, and identify coverage gaps; weeks 5–8 support tactical adjustments through AI-generated narratives, adoption cohorts, and access-barrier escalation; weeks 9–12 recalibrate against benchmarks, shift promotional spend, and record decisions. Databricks Genie lets commercial leaders question unified Rx, specialty-pharmacy, payer-coverage, field-activity, and patient-services data in natural language at prescriber, territory, and regional granularity, with governance and benchmark context. The stated operating benefit is a decision cycle under seven days, enabling teams to detect suppression early, reallocate resources, and respond to access barriers while the launch remains correctable.


### [Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy](https://yomu.fyi/post/scaling-for-mhhs-50x-cost-efficient-margin-data-engineering-at-octopus.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Saad Ali, David Poulet, Daniel Taylor, Ismail Makhlouf
- Published: May 23, 2026

Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.


### [Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks](https://yomu.fyi/post/accelerating-llm-inference-with-prompt-caching-for-open-source-models.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pei-Lun Liao, Asfandyar Qureshi, Roshan Regula, Bruce Fontaine, James Thomas, Chenyang Yu
- Published: May 22, 2026

Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required.


### [Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks](https://yomu.fyi/post/observability-for-any-agent-anywhere-production-ready-tracing-with-ope.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Firas Farah, Bruno Faria, Anoop Sunke
- Published: May 22, 2026

Databricks supports writing OpenTelemetry (OTel) traces directly to Unity Catalog, where real-time telemetry is stored in Delta tables for governed analytics and retention. AI traces capture prompts, tool calls, responses, latency, and execution paths, enabling debugging, evaluation, and monitoring, while lakehouse storage also allows SQL queries, dashboards, joins with business data, and PII controls. The managed, serverless ingestion layer uses Zerobus Ingest to accept OTLP over gRPC from collectors and REST integrations, streaming spans, logs, and metrics to Unity Catalog without intermediate message buses. A LangGraph support manager assistant demonstrates instrumentation with mlflow.langchain.autolog() and an @MLflow.trace root span, while Genie is invoked through MCP for data-driven questions. The resulting traces can be searched in MLflow, evaluated at scale, and monitored continuously, with FAQ details stating 200 QPS starting throughput, no storage limit, and Unity Catalog governance options for access control, masking, and filtering.


### [How World Bank Group uses Databricks to eradicate poverty through shared knowledge](https://yomu.fyi/post/how-world-bank-group-uses-databricks-to-eradicate-poverty-through-shar.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: May 22, 2026

The World Bank Group is building a unified data and AI platform on Databricks to make tens of millions of repository documents and structured operational data findable for teams pursuing shared prosperity. Legacy databases and disconnected document repositories forced researchers to search manually, slowing decisions and limiting the ability to surface lessons learned across its global portfolio. The implementation migrates operational data, applies Unity Catalog governance, uses Databricks Volumes for unstructured content, Genie for natural-language structured queries, a metrics layer for deterministic reporting, and vector search for retrieval-augmented generation over project documents. Because each Genie serves a specific metrics domain, an agentic layer combines intent and domain classification with query decomposition, routing requests among domain agents, RAG, and visualization components before assembling one response; a multi-region feedback prototype was deployed in approximately two and a half days.


### [Using observability data to prevent incidents](https://yomu.fyi/post/using-observability-data-to-prevent-incidents.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Madelyn Mullen
- Published: May 22, 2026

Engineering teams often remain reactive not because they lack telemetry, but because metrics, logs, traces, error budgets, and SLO burn rates are difficult to query at decision speed. The post frames reliability intelligence as trend analysis that identifies accumulating risk before a user-facing incident, contrasting it with observability focused on current system state. It proposes unified telemetry access and natural-language self-service, using Databricks Genie to examine p99 latency, error-budget burn, deployment frequency, incident rates, dependencies, and capacity across services. The Trade Desk’s Genie Router handles questions across more than 10,000 tables, while the post says ad-hoc analysis runs five times faster with Genie and can support mitigation before a page fires.


### [How Databricks Genie democratizes data access in financial services](https://yomu.fyi/post/how-databricks-genie-democratizes-data-access-in-financial-services.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kim Hatton
- Published: May 22, 2026

Financial services organizations have built sophisticated lakehouses, streaming pipelines, model-serving infrastructure, and self-service BI, but access remains concentrated among technical teams. Business leaders still often rely on analysts because they may lack SQL skills, BI training, or analyst access, creating the “last mile” of data democratization. Databricks Genie addresses this gap through a conversational AI interface that converts plain-English questions into governed SQL queries executed against the Databricks Lakehouse without an analyst in the loop. It operates within Unity Catalog access policies, restricts users to authorized data, makes queries read-only, and logs interactions for audit purposes, while its semantic layer maps organizational terminology such as NIM, LTV, and NII to the organization’s meanings. The stated outcome is faster, auditable answers for business questions and usage data that can inform data-product priorities.


[Newer posts](https://yomu.fyi/company/databricks/page/13.md) · [Older posts](https://yomu.fyi/company/databricks/page/15.md)