Loading…
Unity Catalog
112 posts about Unity Catalog. Every summary links to the original.
How the FDA is building a secure, AI-ready data foundation on Databricks for Government
The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.
Filippo Seracini, Vijay RajaOperationalizing Genie Ontology in Your Data Stack
Genie Ontology is presented as a way to give enterprise AI agents shared business context beyond a semantic model, including definitions, relationships, business rules, authoritative sources, and permissions. It combines Unity Catalog Semantics—Metric Views, Pages, and Domains—with context inferred from governed tables, queries, dashboards, notebooks, and other supported assets. The guidance recommends six progressive layers, beginning with clean gold data and resolved golden records, then metadata, semantic modeling, enterprise context, governance, and evaluation. Critical implementation details include declaring informational primary and foreign keys, defining canonical measures in Metric Views, adding synonyms and example queries, and using permissions plus human-reviewed automation. Rather than waiting for complete coverage, it advises starting with one high-value domain and metric, then using feedback, telemetry, benchmarks, and drift reviews to strengthen trust over time.
Srujan Alase, Richard TomlinsonHow Discovery Bank delivers hyper-personalized banking at scale: behavioral AI, governed data, and real-time decisioning
Discovery Bank’s hyper-personalized banking model addresses how to make interactions relevant to individual behavior while meeting financial-services demands for scale, speed, security, and governance. The bank combines demographic, transactional, digital-engagement, savings, borrowing, credit-risk, rewards, and lifestyle-related data on the Databricks Data and AI Platform, producing governed reusable features, indicators, scores, forecasts, and recommendations. Behavioral models and a reusable next-best-action decisioning layer support personalization, fraud detection, servicing, and banker assistance; TRUST alerts assess transactions against client-specific norms and can escalate from explanation to account locking. A four-layer architecture adds control services and specialized generative AI and agents to trusted data and analytical foundations, with governance remaining in the loop. Reported results include a 40% uplift in client engagement impact, 20x faster pipeline development and data processing, 5x faster data-product creation, more than 300 models built per day, and return on investment above 500%.
Jack Yallop, Stuart EmslieAnnouncing the Public Preview of Discover and Domains, powered by Unity Catalog
Databricks announces the Public Preview of Domains and the Discover page, powered by Unity Catalog, to help people and agents identify relevant, high-quality, and safe data and AI assets. Domains organize tables, dashboards, notebooks, queries, metric views, Genie Agents, and apps by business structure, while Discover provides an internal marketplace with search, certification signals, popularity and trending indicators, and AI-powered recommendations. Data stewards can create domains and subdomains, certify assets, add descriptions and contacts, and curate page sections and pinned content. Domains extend Unity Catalog Semantics and feed Genie Ontology, giving agents business context for narrowing retrieval, prioritizing trusted assets, and interpreting metrics within each user’s existing permissions. The features are available in Public Preview through a Databricks workspace, where organizations can create domains and curate assets.
Stef Bran Melendez, Kelly AlbanoTech builds on AI. Finance protects the margin.
AI-native tech companies must protect unit economics as agents accelerate changes in compute consumption, pricing, and revenue recognition, while gross margins remain below classic software levels. Finance teams built around extracts, spreadsheets, and monthly reconciliation can miss repricing changes, metering errors, and compute-commitment risk. The proposed foundation is an evolving ontology that keeps product, plan, usage, and cost meanings current, with Stripe data entering Unity Catalog through OpenSharing and Lakebase providing transactional Postgres on the lakehouse. Genie One uses that ontology to answer governed, sourced questions about gross margin, consumption revenue at risk, and compute spend, while people retain decision authority. The post describes organizations using Databricks to consolidate reporting, forecasting, workflows, and finance applications, positioning a shared data-and-AI platform as the path from an initial answer to an ongoing finance platform.
Madelyn MullenBuilding a soccer coaching app on Databricks
Coach’s Corner, also called La Pizarra, turns high-frequency soccer tracking data into a bench-side application for replay, tactical analysis, scouting, standings, and agent-generated dossiers. Built as a Databricks App, it ingests NDJSON feeds at 25 frames per second through Auto Loader and Spark Declarative Pipelines, enforcing 46 data-quality expectations across bronze, silver, and gold layers. Liquid clustering supports 1–3-second DBSQL queries, while Lakebase synchronizes gold data to Postgres for millisecond replay reads and separates sequential playback from exploratory analytics. The scouting layer grounds Genie, Vector Search, a Unity Catalog-registered xG model, and an Agent Bricks supervisor in governed data, with Claude calls routed through the Unity AI Gateway, MLflow tracing, and a deterministic fallback. Together, these components are presented as a way to deliver traceable insights within seconds without forcing coaches to interpret raw tables or analysts to relay every result.
Samwel Emmanuel, Sheridan Harris, Andrew Helmreich, Kush Patel, Nick RagoneseMeta’s Spark Muse 1.1 is now available on Databricks, fully governed by Unity AI Gateway
Databricks announces support for Meta’s Muse Spark 1.1 through Model Provider Services (MPS) in Unity AI Gateway, addressing fragmented API keys, access controls, and usage visibility when organizations adopt new models. An MPS is a Unity Catalog securable that stores provider configuration and an encrypted API key, while callers use their Databricks credentials and the gateway attaches the key at request time. The post demonstrates registering Muse Spark through the OpenAI provider type with Meta’s API base URL and Responses API, then governing use with Unity Catalog privileges, model allowlists, policies, rate limits, usage metering, and inference tables. Requests are routed through the gateway, where access and guardrails are applied before reaching Meta; usage, spend, tokens, latency, status codes, and optionally full payloads are recorded for attribution and audit.
Pavithra Rao, Shaotong Li, Martin Grund, Kelly AlbanoUnified context: The missing layer for enterprise AI coworkers
Enterprise AI assistants often produce fluent answers yet fail to improve forecast calls, deal reviews, and operational standups because decision context is scattered across systems, teams, and competing definitions. Genie One addresses this by using a shared context layer spanning Databricks data, documents, SaaS applications, and operational systems, allowing questions and follow-up work to retain business meaning. Genie Ontology organizes terms, metrics, entities, and relationships into a living knowledge graph, learning from data, dashboards, queries, documents, and connected applications while ranking definitions and signals using usage and certified-asset links. Together with Unity Catalog, it applies permissions, certified data, shared definitions, and governance controls to answers, actions, and agents. The stated outcome is faster movement from decision preparation to action, with less manual reconciliation while preserving accuracy and control.
Cynthya Peranandam, Christy MaverHow Unity Catalog managed tables bring interoperability, performance, and unified governance to the Lakehouse
Unity Catalog external access to Unity Catalog managed Delta tables is now in Public Preview, allowing external engines to create, read, and write while governance remains centralized. The announcement addresses the previous multi-engine trade-off: external tables enabled access but lacked managed-table performance optimizations and governance guarantees. Catalog commits coordinate writes through Unity Catalog, making it the source of truth for table state and enabling safe external writes, multi-statement transactions, and auditing of external operations. Predictive Optimization cleans storage, collects query statistics, and selects Liquid clustering columns as query patterns change; the post says these capabilities can deliver up to 50% storage cost savings and 20x faster queries. Support includes Spark, Flink, Starburst, DuckDB, and StreamNative, with open APIs and Delta Kernel extending integrations across Databricks UC and UC OSS.
Alex Jiang, Tathagata “TD” DasAI-Enabled Advisory Services for Higher Education
Higher-education call centers face costly, limited-coverage monitoring of advisor conversations and brittle, slow methods for identifying student concerns from transcripts. The proposed workflow deploys OpenAI Whisper on Databricks Model Serving, applies AI Functions for sentiment, topics, intent, and rubric scoring, and uses Unity Catalog to govern the resulting data. For advisor quality, an LLM-as-a-judge evaluates every transcript against a reference-table rubric, returns a weighted 1–5 overall score and per-criterion scores, and routes flagged calls for targeted QA review instead of random sampling. For student insights, quarterly transcript enrichment feeds an Agent Bricks Knowledge Assistant for cited reasoning over raw calls and a Genie Space for structured trend queries, while LangGraph orchestrates UC SQL functions as tools. Together, these components let non-technical advisors, mentors, and QA managers query student interactions without reaching out to a data SME.
Chad Ammirati, Zach Langford, Nicole WongData-Native AI Agents: Why Agents Must Move to Your Data
Enterprise AI pilots often move data into separate vector databases, SaaS LLMs, or serving layers, creating governance gaps, compounded latency, fragmented costs and observability, and duplicated lifecycle work. The post advocates data-native agents: models, agents, tools, retrieval, and memory run inside the governed data platform, with policy enforced during query planning and computation rather than after responses are produced. It argues that post-hoc controls cannot undo sensitive information encoded in aggregations and can trigger token-burning retry loops. For state and memory, it presents Lakebase, managed PostgreSQL within Databricks, as transactional storage and a shared source of truth for multi-agent swarms. The described platform pattern combines Unity Catalog, Unity AI Gateway, Model Serving, MLflow 3, AI Search, Lakebase, and business-context services, and recommends inventorying workloads already outside the perimeter before closing seams incrementally.
Kaan Kuguoglu, John KarlssonFoundational context: Cross-industry & function-specific accelerators for Lakebase
Databricks presents Lakebase as a fully managed, serverless, standard Postgres database for combining operational and analytical workloads on its Data + AI Platform. The platform separates compute from storage, integrates with the lakehouse through Synced Tables and Lakebase CDF, and uses Unity Catalog for governance; copy-on-write branching and autoscaling to zero are described as core infrastructure primitives. The post showcases partner-built, ready-to-deploy accelerators spanning technology, finance, marketing, sales, supply chain, human resources, customer service, and operations. Examples include PostgreSQL migration assessment, multi-agent Genie orchestration, stateful enterprise agents, autonomous data reliability, governed contact-center intelligence, and project operations management. These offerings package Lakebase patterns into migration controls, domain-specific solutions, and agent frameworks intended to accelerate modernization and reduce transformation complexity.
Amit SinghPractical Guide To Python App Hosting
Python app hosting is presented as both an infrastructure choice and, for data-intensive or AI applications, a data-architecture choice. The guide defines a hosting stack comprising application code, runtime, dependency manager, application server and reverse proxy, and contrasts it with traditional web hosting. It compares shared hosting, VPSs and cloud VMs, PaaS, container platforms and serverless functions by control, operational effort, scalability and workload fit. It notes that long-running data and AI tasks such as model training, vector indexes, cached datasets, ETL jobs and lengthy inference pipelines can exceed practical serverless limits. The conclusion is that public APIs and lightweight web apps allow broad platform choice, while apps accessing governed data, model endpoints or AI agents should be evaluated by where data lives and how access is governed, with Databricks Apps presented as an option when data already resides in a lakehouse.
Databricks StaffIntroducing Feature Views
Feature Views are presented as a framework for creating managed feature pipelines in Databricks, addressing duplicated logic, training/serving skew, backfills, online-store plumbing, and governance for real-time ML. A Feature View defines the source, entity, time-series column, and computation once; Databricks uses it to generate point-in-time-accurate historical data, then materializes pipelines for batch or streaming production use. Registered in Unity Catalog, materialized features receive lineage and access controls, while MLflow records model dependencies and Model Serving retrieves required features without custom lookup code. For Kafka-sourced streaming features, Spark Realtime Mode, Lakebase, and Model Serving support rolling-window updates with reported end-to-end p99 latency of 200ms from event to online availability. The announcement says Feature Views are in Public Preview and can be developed through the Feature Engineering Client SDK and Genie Code, with streaming materialization requiring an Enterprise-tier workspace in a region supporting Lakebase.
Nick Joung, Ian Ackerman, Julia PowellThe ambulatory intelligence gap
Health systems face an ambulatory intelligence gap: access, provider capacity, referral retention, panel management, and financial performance are interconnected, while information remains scattered across disconnected systems. Health Catalyst’s Ambulatory Intelligence combines AI with nearly two decades of healthcare improvement expertise, deploying directly in a customer’s Databricks workspace so sensitive data stays within the health system’s environment. It uses a medallion-based semantic layer, Unity Catalog for governance, Lakebase for low-latency serving, and Genie alongside dashboards to help leaders investigate why metrics change. The solution ships with prebuilt metrics across Access Optimization, Revenue Intelligence, Panel Management, and Referral Insights, plus cross-domain scorecards and configurable terminology and workflows. Reported outcomes from supported improvement work include increased revenue and encounters at Thibodaux Regional, 55,000 closed care gaps at INTEGRIS Health, and higher outpatient visits with fewer cancellations without reschedules at WakeMed; future plans include models based on prior outcomes and agentic capabilities.
Morgan Wilkie, Bryan Smith, Holly Burke, Mary EllenAsk, build, compose: What our 5th Genie Hackathon taught us about Databricks Genie
The fifth Databricks hackathon used Databricks Genie to test three ways of working with governed data: asking, building, and composing. Genie Agents give business users domain-specific natural-language access to curated data, while Genie Code helps analysts create metric views, functions, pipelines, and dashboards inside Databricks. The composition track uses Conversation APIs and a managed MCP server to make Genie an agent tool alongside custom logic, Model Serving, and other MCP servers, with Unity Catalog governing access. Projects included supervisory routing across 190 tables, governance and analytics products built in days, and multi-agent systems such as ShipBob’s overnight operations brief. Across the tracks, the stated lesson is that shared governance and semantic context let business users, builders, and engineers use Genie at different levels without abandoning grounded, permissioned data access.
Shruti Prasanna, Rob BajraNavigating a Synapse Migration to Databricks
The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.
Olga Romanova, Johannes OehlerHow Imperial College London is accelerating dementia research with a modern data platform
Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.
Kacey HertanBarracuda makes security logs conversational with Genie
Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.
Barracuda XDR Engineering TeamAutomatic Upgrades: best practice features for your lakehouse tables
Automatic Upgrades gives Unity Catalog managed tables a hands-off path to adopting best-practice table features without manually auditing thousands of tables or running ALTER TABLE repeatedly. It observes table access over a 100-day window, verifies that every accessing Databricks client supports the feature, skips inactive or externally accessed tables, and applies eligible changes through a lightweight background job. New tables inherit a feature when it becomes a schema default, while explicit table properties take precedence. Supported capabilities include Row Tracking, Automatic Liquid Clustering, Deletion Vectors, Column Mapping, Parquet V2, Catalog Commits, and Checkpoint V2, with benefits spanning query speed, storage and write costs, interoperability, and reliability. Changes are visible in DESCRIBE HISTORY and Catalog Explorer, reversible per table, and not re-enabled after a user disables one.
Elizabeth Bowman, Tom van Bussel
