---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [How the FDA is building a secure, AI-ready data foundation on Databricks for Government](https://yomu.fyi/post/how-the-fda-is-building-a-secure-ai-ready-data-foundation-on-databrick.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Filippo Seracini, Vijay Raja
- Published: Sep 1, 2026

The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.


### [Collaboration makes us all stronger](https://yomu.fyi/post/collaboration-makes-us-all-stronger.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aaron Kobayashi, Mehmet D. Ince, Anurag Srivastava, Alexey Kondratov
- Published: Sep 1, 2026

An external researcher, Mehmet Ince, reported a memory-safety flaw in PostGIS’s address\_standardizer extension, where caller-controlled grammar data indexed a fixed-size array without bounds checking. Because the extension was available to ordinary tenant roles on managed Postgres services, Databricks treated the issue as a platform exposure rather than a third-party problem, while noting that its microVM architecture meant the exploit did not produce cross-customer impact on Lakebase Postgres and Neon. Detection alerted the Neon team during testing, enabling rapid validation and downstream patching while a durable fix was developed and deployed to protect tenants without customer action. The upstream fix initially missed cases; Mehmet supplied the remaining pieces, and the complete correction was returned to PostGIS, with his bounty donated and matched for the project.


### [Operationalizing Genie Ontology in Your Data Stack](https://yomu.fyi/post/operationalizing-genie-ontology-in-your-data-stack.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Srujan Alase, Richard Tomlinson
- Published: Sep 1, 2026

Genie Ontology is presented as a way to give enterprise AI agents shared business context beyond a semantic model, including definitions, relationships, business rules, authoritative sources, and permissions. It combines Unity Catalog Semantics—Metric Views, Pages, and Domains—with context inferred from governed tables, queries, dashboards, notebooks, and other supported assets. The guidance recommends six progressive layers, beginning with clean gold data and resolved golden records, then metadata, semantic modeling, enterprise context, governance, and evaluation. Critical implementation details include declaring informational primary and foreign keys, defining canonical measures in Metric Views, adding synonyms and example queries, and using permissions plus human-reviewed automation. Rather than waiting for complete coverage, it advises starting with one high-value domain and metric, then using feedback, telemetry, benchmarks, and drift reviews to strengthen trust over time.


### [The new Brickbuilder Partner Network tiers for ISVs and Data Providers are here](https://yomu.fyi/post/the-new-brickbuilder-partner-network-tiers-for-isvs-and-data-providers.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Stephen Orban
- Published: Sep 1, 2026

The Databricks Partner Network has launched Bronze, Silver, Gold, and Platinum tiers for ISVs and data providers, with placement based on customer impact, technical excellence, strategic alignment, readiness and enablement, and joint go-to-market activity. Technical Excellence is evaluated through the Partner Well-Architected Framework (PWAF), which provides prescriptive, AI-ready guidance for secure, reliable integrations and serves as the standard for validated architecture. For customers, the tiers signal that a partner solution has met a high bar for architecture, security, compliance, and joint customer success, reducing vetting effort. For partners, each tier defines requirements and a roadmap tied to greater visibility, co-selling, and go-to-market support, while Databricks plans to refine the program and expand benefits over the coming months.


### [How Discovery Bank delivers hyper-personalized banking at scale: behavioral AI, governed data, and real-time decisioning](https://yomu.fyi/post/how-discovery-bank-delivers-hyper-personalized-banking-at-scale-behavi.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jack Yallop, Stuart Emslie
- Published: Sep 1, 2026

Discovery Bank’s hyper-personalized banking model addresses how to make interactions relevant to individual behavior while meeting financial-services demands for scale, speed, security, and governance. The bank combines demographic, transactional, digital-engagement, savings, borrowing, credit-risk, rewards, and lifestyle-related data on the Databricks Data and AI Platform, producing governed reusable features, indicators, scores, forecasts, and recommendations. Behavioral models and a reusable next-best-action decisioning layer support personalization, fraud detection, servicing, and banker assistance; TRUST alerts assess transactions against client-specific norms and can escalate from explanation to account locking. A four-layer architecture adds control services and specialized generative AI and agents to trusted data and analytical foundations, with governance remaining in the loop. Reported results include a 40% uplift in client engagement impact, 20x faster pipeline development and data processing, 5x faster data-product creation, more than 300 models built per day, and return on investment above 500%.


### [Autoscaling Lakebase Postgres](https://yomu.fyi/post/autoscaling-lakebase-postgres.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Carlota Soto
- Published: Aug 31, 2026

Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.


### [What is an AI Copilot?](https://yomu.fyi/post/what-is-an-ai-copilot.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 28, 2026

An AI copilot is an assistant embedded in a software application that uses workflow context to offer suggestions, generated content, or approved actions while keeping the human in control. Unlike a standalone chatbot, it combines an LLM with contextual grounding, retrieval-augmented generation (RAG), access to enterprise data, and action layers that can call APIs, execute code, trigger workflows, or update records. The explainer surveys code, productivity, data and analytics, customer-facing, and domain-specific copilots, including uses such as SQL generation, dashboard creation, email drafting, and case summarization. Reported benefits include faster routine work, broader data access, more consistent outputs, and less context switching; a cited GitHub study found Copilot users completed a controlled coding task 55% faster. It also stresses hallucinations, privacy, bias, overreliance, integration complexity, cost, and the requirement for human review, governance, and accountability.


### [Beyond answers: New Genie One features to turn insights into action](https://yomu.fyi/post/beyond-answers-new-genie-one-features-to-turn-insights-into-action.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elise Georis, Rhetta Nadas, Sydney Sundell
- Published: Aug 28, 2026

Genie One’s latest updates target the manual work that follows an AI-generated answer, including accuracy checks, report formatting, sharing, and repeated workflows. The release adds a native macOS desktop app in beta with a global launcher, persistent conversations across web and desktop, and access to documents, skills, scheduled tasks, Genie Agents, and Databricks Apps. Genie One can create editable documents with live visualizations, comments, version history, sharing, PDF export, and external-document editing through MCP writes. Users can also turn conversations into reusable, shareable Genie Agents or distribute read-only chat links containing questions, answers, visualizations, and context. Additional capabilities include ontology snippets, private uploads of supported files, and governed MCP actions, with Unity Catalog, source authentication, and Unity Gateway governing access across assets and connected tools.


### [How Trackunit turns construction data into decisions with AI](https://yomu.fyi/post/how-trackunit-turns-construction-data-into-decisions-with-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Domokos Spéder, Erin Kirsten, Jack Yallop
- Published: Aug 28, 2026

Construction data is fragmented across systems, organizations, equipment types, and formats, limiting the operational context available for decision-making. Trackunit addresses this data-intelligence problem with IrisX, an operating data platform built on Databricks that connects equipment, machine, operator, site, and operational data. Its three-part model—connect, distill, and amplify—combines data engineering, analytics, and AI on an open platform, then delivers governed, construction-specific intelligence through applications, workflows, and AI interfaces. The platform supports natural-language analysis of equipment signals and contextual factors such as maintenance history, operating conditions, location, contracts, project needs, and asset availability. Examples described include battery-management guidance for OEMs, missed-invoice detection for rental companies, and asset redeployment for contractors, with reported value of approximately $3 million and $2 million in two examples.


### [How Indra unified EV charging data on Databricks](https://yomu.fyi/post/how-indra-unified-ev-charging-data-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jack Yallop
- Published: Aug 28, 2026

Indra Renewable Technologies consolidated a fragmented EV-charging data estate spanning Cosmos DB, Synapse, Azure Data Lake Storage, Azure Functions, Power BI, and other services onto Databricks. The migration began with a parallel Synapse prototype and expanded to Delta tables, a consolidated fleet pipeline, and greater use of Databricks for operational workloads. A medallion architecture stores raw fleet data in bronze, transforms it with PySpark in silver, and publishes gold tables for reporting and client delivery; one pipeline now serves three clients and improved performance by 60% to 70%. Unity Catalog, serverless SQL warehouses, AI/BI dashboards, and Genie Agents support governed self-service analytics, while reported results include 80% to 90% business cost savings, 90% storage savings, and query latency falling from 24.4 seconds to 3.5 seconds.


### [Fast, fault-tolerant PyTorch training on AI Runtime](https://yomu.fyi/post/fast-fault-tolerant-pytorch-training-on-ai-runtime.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Bruce Fontaine
- Published: Aug 28, 2026

At scale, training jobs face frequent GPU interruptions, while remote-storage input pipelines can leave accelerators idle. Distributed Checkpoint (DCP) has each rank write a shard plus a .metadata file, allowing parallel saves and reloads onto a different number of GPUs; DCP is presented as useful for DDP as well as sharded training. Asynchronous saves stage data locally and upload in the background; cited measurements report 1.8x savings for a 2.8B-parameter DDP model and 58x for a 20B-parameter FSPD model on 32 H100s, though the comparison excludes torch.save network storage time. UCVolumeDataset and the accompanying DataLoader cache files on local NVMe and prefetch disjoint rank/worker slices while computation runs. Correct recovery also requires checkpointing data position plus seeds and RNG state so resumed sampling remains consistent.


### [Building for the AI Era: Lakebase, Streaming, and Lakehouse Innovations  at VLDB 2026](https://yomu.fyi/post/building-for-the-ai-era-lakebase-streaming-and-lakehouse-innovations-a.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Indrajit Roy, Ippokratis Pandis
- Published: Aug 27, 2026

Databricks previews its VLDB 2026 keynote, four accepted papers, a demo, and a sponsor talk covering database architecture, streaming, and Lakehouse optimization. Reynold Xin will introduce Lakebase, which separates serverless PostgreSQL compute from storage, and LTAP, a transactional-analytical model intended for AI-agent workloads; Lakebase stores data and write-ahead logs in cloud object storage and supports sub-second cold starts and copy-on-write branching. The papers describe a decade of Spark Structured Streaming evolution, including microbatch pipelining that improved throughput by up to 3x, stateful APIs, and fine-grained access control, alongside AutoLiquid’s CLUSTER BY AUTO and Ultron’s history-based query optimization. AutoLiquid outperformed customer-selected clustering keys on over 95% of evaluated workloads, while Ultron improved median join latency by 25% in production workloads. The program also includes an Enzyme demonstration of incremental materialized-view maintenance and a sponsor presentation on LakehouseRT and the Reyden engine.


### [What QSR reports miss about the decisions matter the most](https://yomu.fyi/post/what-qsr-reports-miss-about-the-decisions-matter-the-most.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Dedra Berg, Omid Krabbe, Yared Gudeta
- Published: Aug 27, 2026

Traditional QSR reporting can show that a limited-time offer (LTO) missed systemwide expectations, but not whether the cause was weak demand, lower franchise participation, ingredient shortages, inconsistent execution, slower service, or menu cannibalization. The post presents Lovelytics’ QSR Executive Performance Control Tower, an AI blueprint/solution built on Databricks that connects corporate performance, supply chain and operations, customer and digital, and financial signals. It is designed to distinguish demand from operational and participation issues, link changes to likely drivers and financial impact, and give corporate and franchise leaders a daily briefing on what changed, why it matters, and where to focus. The stated advantage is faster, more actionable decision-making based on a governed, reusable intelligence layer rather than disconnected reports.


### [Enhancing Agent Retrieval with Structured Chart Extraction](https://yomu.fyi/post/enhancing-agent-retrieval-with-structured-chart-extraction.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: Aug 27, 2026

Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai\_parse\_document. Its pipeline uses ai\_prep\_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines.


### [Vertical Advantage: Transforming Industries with Lakebase and Agentic AI](https://yomu.fyi/post/vertical-advantage-transforming-industries-with-lakebase-and-agentic-a.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amit Singh
- Published: Aug 27, 2026

The post presents partner-built, industry-specific solutions on Databricks Lakebase for financial services, manufacturing and energy, retail and consumer sectors, travel and hospitality, healthcare and life sciences, communications and media, and the public sector. Lakebase is described as a serverless Postgres transactional database on the Databricks Data and AI Platform, combining operational and analytical workloads under unified governance and providing operational serving, agent memory, audit trails, and transactional runtime capabilities. Examples include regulation-change monitoring, AI-native claims operations, KYC and personalization accelerators, near-real-time fraud detection, telecom customer-lifecycle decisioning, and public-sector emergency Situation Reports, using components such as Unity Catalog, Lakeflow, Genie, Mosaic AI agents, and Agent Bricks. The stated outcome is production-ready acceleration from insight to action, including faster compliance response, claims adjudication in minutes, low-latency decisions, and reduced manual effort, with partners positioned to help scope pilots.


### [Object Storage + WAL: Lakebase Postgres for the agentic era](https://yomu.fyi/post/object-storage-wal-lakebase-postgres-for-the-agentic-era.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Cassie Murray, Carlota Soto
- Published: Aug 27, 2026

The post presents Lakebase Postgres as a design for agent workloads that need isolated copies, restores, replicas, historical queries, and many short-lived databases without repeatedly moving large datasets. It contrasts conventional data-centric OLTP with a transaction-centric model in which PostgreSQL’s write-ahead log (WAL) is the source of truth, with LSNs making database history addressable. Lakebase splits compute, which runs standard Postgres and uses RAM and local NVMe caches, from storage, where safekeepers replicate WAL through quorum acknowledgements, a pageserver materializes pages, and object storage retains immutable history. Commits wait for replicated WAL acknowledgement, not page uploads, while reads request pages at an LSN. The design supports point-in-time queries, cheap branch or read-only compute, scale-to-zero suspension, and shared transactional and analytical access, while still relying on RAM and NVMe for performance.


### [Introducing Governance Hub: Intelligent, account-level governance over your Databricks estate](https://yomu.fyi/post/introducing-governance-hub-intelligent-account-level-governance-over-y.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sadhana Bala, Kristen Wilder, Jodi Boone
- Published: Aug 26, 2026

Databricks introduces Governance Hub, a beta account-level interface for platform and governance teams overseeing estates spanning hundreds of workspaces and multiple regions. It consolidates data health, access, AI usage, and cost information across AWS, Azure, and GCP, replacing scattered system tables, workspace views, and third-party tools with drill-downs for metadata gaps, principal access, token and model activity, and untagged spend. The hub also integrates with Genie, allowing users to ask governance questions in natural language and receive answers and recommendations grounded in governance data; future actions such as configuring policies and alerts are described as planned. Account admins enable it from the Account Console, with access varying by role and existing permissions respected, while upcoming features include performance and security insights and delegated recommendations.


### [Leveraging Databricks to Support FISC Security Guidelines](https://yomu.fyi/post/leveraging-databricks-to-support-fisc-security-guidelines.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jarrett Hoffman, Abhi Arikapudi
- Published: Aug 26, 2026

The FISC Security Guidelines define control objectives for Japanese financial institutions, while the post addresses the challenge of translating them into auditable controls on Databricks data and AI workloads. The Databricks Data + AI Platform, Unity Catalog, and optional Enhanced Security Monitoring and Compliance Security Profile features are presented for identity and network controls, encryption, fine-grained access, audit logs, lineage, and hardened runtimes. For AI workloads, Unity Catalog’s Model Registry links training data, code versions, and evaluation metrics to model versions, while a separate framework maps 97 AI-specific risks to 73 controls. The FISC Customer Capabilities Mapping Matrix connects guideline units to platform capabilities and customer responsibilities, including configuration, SIEM integration, backup, recovery, and evidence; the Security Analysis Tool is suggested for finding workspace gaps, and the material is identified as educational guidance requiring expert review.


### [How Capital Markets Finance Protects Balance Sheet Returns](https://yomu.fyi/post/how-capital-markets-finance-protects-balance-sheet-returns.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andrea DeSosa, Kim Hatton
- Published: Aug 25, 2026

Capital markets finance must protect returns on a balance sheet exposed continuously to market, funding, and liquidity risk, while funding costs, capital consumption, and liquidity needs can change intraday. Fragmented desk P&L, treasury, and risk data require shared business context, with ontology and governance linking numbers to desks, products, positions, funding, capital, and permissions. Databricks’ Genie One is presented as a data-smart AI coworker that lets authorized finance users investigate governed data, trace emerging funding or profitability issues, and surface the definitions and source evidence behind capital and valuation figures. One top global bank reduced regulatory data processing for liquidity reporting from 10 hours to 8 minutes, while pricing, funding, risk, and capital decisions remain with accountable people.


### [Choosing Data Governance Tools for Enterprise Data Governance](https://yomu.fyi/post/choosing-data-governance-tools-for-enterprise-data-governance.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 25, 2026

Enterprise data governance tools address fragmented data estates in which teams struggle to locate assets, identify ownership, protect sensitive information, and maintain consistent quality across warehouses, lakes, SaaS applications, and spreadsheets. Positioned above storage and compute, these tools read metadata, schemas, and query logs to provide a shared governance layer for cataloging, lineage, classification, policy enforcement, quality monitoring, and audit reporting. The guide distinguishes standalone catalogs, point solutions, enterprise suites, platform-native tools, and open-source options while treating AI and agent governance as an increasingly important capability for models, prompts, and autonomous agents. It recommends evaluating candidates against seven criteria, comparing total cost of ownership, and planning a phased rollout that starts with highest-risk domains; a lakehouse-native approach can reduce synchronization between separate governance systems.


[Older posts](https://yomu.fyi/company/databricks/page/2.md)