Loading…
Unity Catalog
112 posts about Unity Catalog. Every summary links to the original.
Reimagining Data Modeling on the Lakehouse: Introducing Vibe Data Modeling
Vibe Data Modeling is presented as a way to build a governed Silver-layer data model from a plain-English description of a business, addressing the months or years often required to hand-build one or adapt a generic industry template. The single-notebook implementation uses four widgets and a multi-model LLM agent to organize organizations into divisions, domains, subdomains, products, and attributes, then generate a logical model and deploy it to Unity Catalog. Its pipeline advances through four generate-and-validate stages, applying 251 rules, deterministic structural gates, two architect reviews, and a retry loop that changes strategy when checks fail. The authoritative model.json also produces schemas, Delta tables, informational foreign keys, classification tags, metric views, an RDFS ontology, DBML, and synthetic sample data, while plain-English refinements create auditable, reversible versions.
Amr Ali, Cary Moore, Roberto Bruno Martins, Abhijit TilakOpenAI and Databricks at DAIS 2026: Making enterprise AI real
At Data + AI Summit 2026, Databricks and OpenAI presented a partnership centered on combining OpenAI’s frontier models and agents with Databricks’ enterprise context and control. The post describes GPT models and Codex running natively on Databricks, with Unity AI Gateway governing interactions, budgets, routing, auditing, and visibility, while Agent Tools provide governed enterprise data access through MCPs. Examples include OpenAI’s marketing data foundation on Databricks, which cut storage costs by $400,000 per month, and a Hertz application built in 11 business days with GPT-5.5 and Databricks that lifted lead conversion from 60–65% to 75–80%. The sessions argue that deployment, security, evaluation, monitoring, context, and sharing comprise most of the work around enterprise agents. A joint virtual event scheduled for August will address shipping agentic applications at scale, including a Stellantis case study.
Margaret AmoriBeyond dashboards: Introducing Decision Execution Platforms
Databricks Forward Deployed Engineering introduces Decision Execution Platforms (DEPs), an enterprise analytics category intended to connect KPI signals, executive decisions, operational execution, and measured outcomes. The proposal addresses workflows in which dashboards reveal problems but meetings, decks, spreadsheets, and messaging threads leave implementation fragmented and impact measurement disconnected. DEPs run the four-stage loop on governed Databricks infrastructure: agents recommend actions, alternatives, predicted impact, and reasoning; approved choices execute through systems of record; and results persist in a Decision Log for continuous learning. Their architecture combines a foundation of Lakebase, Genie, Unity Catalog, Lakehouse, Agent Bricks, and MLflow with an SDK of reusable primitives and a Databricks Apps executive surface. A retailer case used a DEP to unify fulfillment data and enable simulated, controlled rerouting, with scaling aimed at measurable bottom-line and customer-satisfaction outcomes.
Marc Solomon, Marcello PedersenA Decision Framework for ETL Migration to Databricks
An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.
Rafael AieloHow the English Office for Students leverages Databricks to enhance higher education standards and drive better student outcomes
The Office for Students, which regulates more than 400 higher education providers in England, modernised its data and analytics environment after a legacy platform could no longer handle growing volumes, varied sources, or emerging analytical demands. Its data spans millions of student records collected over 15 to 20 years, and a workflow processing about 300 million records took eight hours. Moving to Databricks consolidated structured, qualitative, and near-live data with analytics and AI workflows, while Unity Catalog added lineage, access controls, and security patterns for governed use. Genie Code reduced a student segmentation analysis from at least two weeks for two analysts to half a day, and a provider-registration triage proof of concept flags missing submissions earlier. The organisation frames AI as decision support rather than decision-making, keeping humans responsible for regulatory judgments.
Kacey HertanFrom test bench to lakehouse: how AVL modernizes measurement data analytics with Impulse
AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.
Dr. Thomas Bonfert, Jonathan Bräuer, Fabian Ade, Maxim Hammer, Florian Gorzitzke, David Crescence, Christa Simon, Jörg Zimmermann, Hannes SchneiderWhat To Look For in a Serverless Database for AI Applications
Serverless databases are presented as a baseline for AI applications whose traffic can be volatile, idle for long periods, or spike when agents fan out queries. The guide distinguishes managed serverless systems from autoscaling products by focusing on architectural separation of compute and storage, true scale-to-zero, cold-start behavior, connection handling, pricing, performance, portability, governance, and AI capabilities such as vector search. It recommends evaluating both low- and high-utilization costs, published warm-up times, tail latency (p95/p99), and built-in pooling or HTTP/Data APIs for high-concurrency agents and serverless functions. The text positions Lakebase as an example that combines serverless Postgres, shared lakehouse storage, and Unity Catalog governance, and cites reported cost and management reductions from a 2025 study while noting that provisioned deployments may suit continuously high-throughput workloads.
Databricks StaffThe Rise of Sports Intelligence: How the Lakehouse Turns Tracking Data into Competitive Advantage
Professional basketball’s Hawk-Eye SkeleTRACK feed produces roughly 22,620 positional updates per second—about 65 million records per 48-minute game—yet teams often cannot turn that volume into timely, trusted decisions. The integration gap comes from separate vendors for tracking, wearables, video, scouting, and medical data, alongside calibration differences, weak provenance, and compute limits. The Databricks Data + AI Platform is presented as a governed lakehouse that ingests feeds with Lakeflow, refines them through medallion layers, and uses Unity Catalog for lineage, access control, and auditing. Models for shot probability, injury risk, and fatigue can run alongside serving and custom applications, with Lakebase supporting sub-second interactive queries. Applications include proactive load management, real-time coaching intelligence, and enriched broadcast or fan experiences across tracking-rich sports.
Corey Abshire, Kush Patel, Nick RagoneseHow Daikin Applied Americas builds consistent data pipelines at scale with Genie Code
Daikin Applied Americas needed to scale reliable data pipelines across growing analytics and AI use cases involving operational, manufacturing, and service data while coordinating development across teams. It adopted Databricks Genie Code within a structured operating model, using Unity Catalog context, reusable MECE skills, and explicit checkpoints across Bronze, Silver, and Gold layers to guide planning and execution. The framework defines competencies such as source grain, transformation patterns, canonical alignment, governance, and business-entity modeling, moving standards out of long prompts and into the development environment. The team reports that pipelines that once took days to prototype could be generated in minutes, with faster iteration, more consistent outputs, less structural correction, reduced architectural drift, and greater trust in AI-assisted results.
Trent Lezer, James VanGordonWhat if the answer was already in your data?
Kythera Labs is building an AI-native healthcare strategy platform on Databricks to give health systems access to strategic intelligence that historically required specialized analysts or consulting firms. Its foundation converts 339 billion medical and prescription claims covering more than 300 million patients into governed, event-based data, resolving providers, harmonizing codes across 130 vocabularies, and reconstructing patient journeys. Healthcare Strategy Agent, built with Agent Bricks, lets executives ask questions such as where oncology referrals are going and receive analyses of leakage, competing providers, physicians, and reimbursement opportunity in minutes. A Louisiana health system went live within ten days and reported 150% greater visibility into encounters, 12% more keepage, 22% less leakage, and $3.8 million in estimated annualized retained-encounter value. Unity Catalog, Lakebase, Delta Lake, Delta Sharing, and serverless infrastructure provide shared governance, lineage, access controls, and operational integration.
Bryan Smith, Morgan Wilkie, Kaitlin RyanDatabricks positioned highest in execution and furthest in vision for the second consecutive year in Gartner Magic Quadrant
Databricks says Gartner positioned it highest for execution and furthest for vision in the Magic Quadrant for the second consecutive year. The post connects this recognition to a category reclassified from “Data Science and Machine Learning” to “AI Platforms for Data Science and Machine Learning,” and argues that agentic applications require enterprise data, governance, observability, and business context. Databricks presents a unified approach combining the lakehouse, Lakebase, Agent Bricks, Unity Catalog, and Unity AI Gateway to build, monitor, and govern agents, models, data, apps, and tools. Reported examples include YipitData’s 20x increase in company coverage with 92–95% tagging accuracy, Block’s unified AI and data estate, and Novo Nordisk’s attribution of more than $157 million in net new value to governed clinical-trial optimization.
Craig Wiley, Kasey Uhlenhuth, Kayli Berlin, Cynthya PeranandamGenesis Workbench: A blueprint for industry AI in life sciences, powered by Databricks and NVIDIA
Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.
Mark Lee, Srijit NairData scientists: Powering the future of AI and analytics
Data scientists connect statistics, programming, and domain knowledge to turn raw data into models, recommendations, and decisions with business consequences. The role now extends beyond classical modeling to large language models, generative AI applications, agentic systems, and production workflows involving deployment, monitoring, and retraining. Modern practice combines Python, SQL, and libraries such as scikit-learn, PyTorch, Spark, and MLflow with data engineering basics, communication, and domain expertise. Data scientists contribute across problem framing, data access, preparation, feature engineering, experimentation, deployment, and lifecycle monitoring, with governed access and lineage helping align training data with production data. The article concludes that AI assistants and agents can automate routine work, but human judgment remains necessary for trustworthy evaluation, business framing, and actionable recommendations.
Databricks StaffHow Stagwell built privacy-safe ID matching on Databricks
Stagwell built a Databricks Marketplace App to let brands match first-party records against its Identity Spine without sending raw customer data outside their own workspaces. It combines Databricks Clean Rooms, Unity Catalog, Jobs and Notebooks, Delta Sharing, and a React/Express interface. Workspace OAuth and On-Behalf-Of authorization enforce the user's Unity Catalog permissions, while separate service principals and a scoped personal access token support app and clean-room operations. Within the clean room, a packaged notebook joins brand data with the Identity Spine, resolves multiple identifiers, and computes match rates, coverage metrics, and household and consumer IDs; results include aggregated audience insights for activation, while raw customer data and proprietary matching code remain separated.
Sridhar Sundaresan, Suvan KaulWhat is Row-Level Security?
Row-level security (RLS) is a database access control that filters which table rows a user may read or change according to identity, role, or session context. It supports multi-tenant SaaS, regional segregation, departmental access, compliance, healthcare, and partner use cases, while typically layering table-level grants with column-level security or data masking. The database applies a policy or predicate automatically at query time across BI dashboards, notebooks, ad-hoc SQL, APIs, and third-party tools, with separate USING and WITH CHECK rules for reads and writes. On Databricks, Unity Catalog row filters use SQL user-defined functions and apply across Databricks SQL, notebooks, jobs, and connected BI tools. The source recommends simple predicates, indexed keying columns, identity-based testing, and explicit enforcement for both reads and writes.
Databricks StaffDatabricks and NVIDIA: Building for the Agentic Era
Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.
Hanlin Tang, Tejas SundaresanIntroducing the Agentic CDP: A New Species of CDP for a New Era of Agents
Traditional customer data platforms (CDPs) were built for human-managed, batch-based campaigns, but the post argues that agentic buying requires millisecond speed, hyper-personalization, and richer context. It contrasts the familiar Golden Record with Golden Context, which combines customer data with current business goals and the history and outcomes of prior decisions. The proposed Agentic CDP uses “Infinity Campaigns,” always-on engagement loops that use LLMs and agents to adapt messaging, timing, and channels for individuals. It is also embedded in the data foundation, bringing customer, business, and decision context together under existing governance, and is designed for agents and humans from the outset. Databricks presents CustomerLake as an implementation of these principles for its platform.
Tasso Argyros, Ali Ghodsi, Reynold XinEnabling Governed Vibe Coding for Enterprise Apps on Databricks
Databricks introduces three capabilities intended to bring vibe coding to enterprise applications, where speed alone does not provide business-data context, deployment safety, or cost control. App Spaces lets admins define resource and data access, on-behalf-of-user API scopes, and security policies for groups of apps, with each app inheriting those settings. Genie App Builder turns plain-language descriptions into working internal apps through generated plans, live previews, AppKit, and awareness of workspace data assets and Unity Catalog semantics. Serverless micro apps run in isolated lightweight virtual machines, start quickly when needed, scale to zero while idle, and use usage-based rather than reserved-capacity infrastructure. Together, the capabilities are presented as a way for business-proximate users to build on enterprise data while organizations apply consistent governance and support broader app portfolios; all three are coming to Databricks Apps, with private previews coming soon.
Evan Pandya, Justin DeBrabant, Cong XuIntroducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
Huey Han, Harish Gaur, Akram Chetibi, Mengxi ChenAnnouncing Apps on Databricks Marketplace
Databricks announces the Public Preview of Apps on Databricks Marketplace, allowing customers to discover, install, and run third-party data and AI applications inside secure Databricks workspaces. The offering addresses procurement challenges involving data movement, lengthy security reviews, custom integrations, and fragmented identity management by bringing applications to the customer’s data. Installed apps run in isolated sandboxes within the consumer’s Databricks account, inherit Unity Catalog governance, and use dedicated serverless compute with consumer-controlled external access through Serverless Egress Gateway policies. Providers can publish closed-source containerized apps once for no-egress distribution without maintaining per-customer infrastructure, while applications connect natively to services including SQL Warehouse, Lakebase, Model Serving, and Foundation Model APIs. The Public Preview launches with 20 partners, and planned additions include bundled assets, provider analytics, and commercial monetization.
Tia Chang, Akram Chetibi, Harish Gaur, Stephen Orban, Mengxi Chen
