Loading…
Topic
136 posts about Unity Catalog. Every summary links to the original.
See the idea
An agentic ad stack wires governed customer signals to campaign execution through a shared tool interface and server-side guardrails. The diagram shows each layer, the direction of a tool call, and where control stays human.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Enterprise data products are presented as managed assets that deliver the right data at the right time, with appropriate quality, format, and usability for users and applications. Their value depends on clear ownership, business-aware data product owners, and five trustworthiness characteristics: quality and observability, semantic consistency, privacy, security, and discoverability. The proposed lifecycle runs from inception and design through creation, publication, operation and governance, consumption, and retirement, with data contracts, quality metrics, testing, versioning, access management, and lineage supporting those stages. For Databricks implementations, the guidance recommends Delta Live Tables expectations and constraints, staged development, testing and production environments, automated monitoring, and Unity Catalog for discovery, governance, lineage, auditing, and access controls.
Southern Company’s SCOUT is a real-time storm operations application designed to make outage intelligence accessible beyond control-center desks during active events. Built on the Databricks lakehouse, it combines Delta Lake storage, Databricks Lakehouse warehouses, Unity Catalog governance, and collaborative notebooks, querying a dedicated warehouse every minute for current operational data. The mobile-friendly application unifies outage, customer, GIS, weather, terrain, crew, and reliability information, helping teams assess restoration times, assign appropriately equipped crews, coordinate mutual assistance, and communicate with customers. SCOUT is used across Southern Company’s operating businesses and had 1,139 adopted employees, including more than 250 users on a peak storm day; the source also describes exploration of AI crew recommendations and autonomous-drone integration, not a completed deployment.
Genie Agents updates expand expert-curated business agents beyond quick factual lookups to deeper analysis across structured and unstructured data. Agent mode can create and refine research plans, issue iterative queries, and return reports with findings, visualizations, and citations; Agent mode APIs extend this workflow to custom applications, chatbots, scheduled reports, and internal tools through streaming Server-Sent Events. Agents can analyze PDFs, documents, slide decks, and images in Unity Catalog volumes alongside governed tables, with permissions and content search indexing applied to retrieval. Genie Code helps authors create baseline agents, diagnose conversation or benchmark failures, and identify knowledge gaps from user feedback, while authors retain review control over proposed changes. Shared chats keep follow-up messages and edited visualizations visible to authorized users, supporting collaboration and agent-quality review.
The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.
Genie Ontology is presented as a way to give enterprise AI agents shared business context beyond a semantic model, including definitions, relationships, business rules, authoritative sources, and permissions. It combines Unity Catalog Semantics—Metric Views, Pages, and Domains—with context inferred from governed tables, queries, dashboards, notebooks, and other supported assets. The guidance recommends six progressive layers, beginning with clean gold data and resolved golden records, then metadata, semantic modeling, enterprise context, governance, and evaluation. Critical implementation details include declaring informational primary and foreign keys, defining canonical measures in Metric Views, adding synonyms and example queries, and using permissions plus human-reviewed automation. Rather than waiting for complete coverage, it advises starting with one high-value domain and metric, then using feedback, telemetry, benchmarks, and drift reviews to strengthen trust over time.
Discovery Bank’s hyper-personalized banking model addresses how to make interactions relevant to individual behavior while meeting financial-services demands for scale, speed, security, and governance. The bank combines demographic, transactional, digital-engagement, savings, borrowing, credit-risk, rewards, and lifestyle-related data on the Databricks Data and AI Platform, producing governed reusable features, indicators, scores, forecasts, and recommendations. Behavioral models and a reusable next-best-action decisioning layer support personalization, fraud detection, servicing, and banker assistance; TRUST alerts assess transactions against client-specific norms and can escalate from explanation to account locking. A four-layer architecture adds control services and specialized generative AI and agents to trusted data and analytical foundations, with governance remaining in the loop. Reported results include a 40% uplift in client engagement impact, 20x faster pipeline development and data processing, 5x faster data-product creation, more than 300 models built per day, and return on investment above 500%.
Rippling describes an architecture in which Databricks writes some tables while Trino, Snowflake, and other lakehouse engines read them, making catalog ownership a design decision rather than a metadata detail. Its rule is that the catalog follows the writer: AWS-native workloads use Glue, Databricks workloads use Unity Catalog, and Snowflake-produced shared data uses Horizon Catalog with Snowflake-managed Iceberg tables. For selected Databricks outputs, Unity Catalog-managed Apache Iceberg stores table data and metadata in Rippling-owned S3, exposes access through the Iceberg REST Catalog API or federation, and uses credential vending for scoped storage access. This lets downstream engines read Parquet directly without duplicate materialization or producer-side query execution. Databricks also provides Predictive Optimization for maintenance, while the trade-off is continued Unity Catalog dependence for metadata, permissions, lineage, and lifecycle.
Databricks announces the Public Preview of Domains and the Discover page, powered by Unity Catalog, to help people and agents identify relevant, high-quality, and safe data and AI assets. Domains organize tables, dashboards, notebooks, queries, metric views, Genie Agents, and apps by business structure, while Discover provides an internal marketplace with search, certification signals, popularity and trending indicators, and AI-powered recommendations. Data stewards can create domains and subdomains, certify assets, add descriptions and contacts, and curate page sections and pinned content. Domains extend Unity Catalog Semantics and feed Genie Ontology, giving agents business context for narrowing retrieval, prioritizing trusted assets, and interpreting metrics within each user’s existing permissions. The features are available in Public Preview through a Databricks workspace, where organizations can create domains and curate assets.
AI-native tech companies must protect unit economics as agents accelerate changes in compute consumption, pricing, and revenue recognition, while gross margins remain below classic software levels. Finance teams built around extracts, spreadsheets, and monthly reconciliation can miss repricing changes, metering errors, and compute-commitment risk. The proposed foundation is an evolving ontology that keeps product, plan, usage, and cost meanings current, with Stripe data entering Unity Catalog through OpenSharing and Lakebase providing transactional Postgres on the lakehouse. Genie One uses that ontology to answer governed, sourced questions about gross margin, consumption revenue at risk, and compute spend, while people retain decision authority. The post describes organizations using Databricks to consolidate reporting, forecasting, workflows, and finance applications, positioning a shared data-and-AI platform as the path from an initial answer to an ongoing finance platform.
Coach’s Corner, also called La Pizarra, turns high-frequency soccer tracking data into a bench-side application for replay, tactical analysis, scouting, standings, and agent-generated dossiers. Built as a Databricks App, it ingests NDJSON feeds at 25 frames per second through Auto Loader and Spark Declarative Pipelines, enforcing 46 data-quality expectations across bronze, silver, and gold layers. Liquid clustering supports 1–3-second DBSQL queries, while Lakebase synchronizes gold data to Postgres for millisecond replay reads and separates sequential playback from exploratory analytics. The scouting layer grounds Genie, Vector Search, a Unity Catalog-registered xG model, and an Agent Bricks supervisor in governed data, with Claude calls routed through the Unity AI Gateway, MLflow tracing, and a deterministic fallback. Together, these components are presented as a way to deliver traceable insights within seconds without forcing coaches to interpret raw tables or analysts to relay every result.
Databricks announces support for Meta’s Muse Spark 1.1 through Model Provider Services (MPS) in Unity AI Gateway, addressing fragmented API keys, access controls, and usage visibility when organizations adopt new models. An MPS is a Unity Catalog securable that stores provider configuration and an encrypted API key, while callers use their Databricks credentials and the gateway attaches the key at request time. The post demonstrates registering Muse Spark through the OpenAI provider type with Meta’s API base URL and Responses API, then governing use with Unity Catalog privileges, model allowlists, policies, rate limits, usage metering, and inference tables. Requests are routed through the gateway, where access and guardrails are applied before reaching Meta; usage, spend, tokens, latency, status codes, and optionally full payloads are recorded for attribution and audit.
Enterprise AI assistants often produce fluent answers yet fail to improve forecast calls, deal reviews, and operational standups because decision context is scattered across systems, teams, and competing definitions. Genie One addresses this by using a shared context layer spanning Databricks data, documents, SaaS applications, and operational systems, allowing questions and follow-up work to retain business meaning. Genie Ontology organizes terms, metrics, entities, and relationships into a living knowledge graph, learning from data, dashboards, queries, documents, and connected applications while ranking definitions and signals using usage and certified-asset links. Together with Unity Catalog, it applies permissions, certified data, shared definitions, and governance controls to answers, actions, and agents. The stated outcome is faster movement from decision preparation to action, with less manual reconciliation while preserving accuracy and control.
Unity Catalog external access to Unity Catalog managed Delta tables is now in Public Preview, allowing external engines to create, read, and write while governance remains centralized. The announcement addresses the previous multi-engine trade-off: external tables enabled access but lacked managed-table performance optimizations and governance guarantees. Catalog commits coordinate writes through Unity Catalog, making it the source of truth for table state and enabling safe external writes, multi-statement transactions, and auditing of external operations. Predictive Optimization cleans storage, collects query statistics, and selects Liquid clustering columns as query patterns change; the post says these capabilities can deliver up to 50% storage cost savings and 20x faster queries. Support includes Spark, Flink, Starburst, DuckDB, and StreamNative, with open APIs and Delta Kernel extending integrations across Databricks UC and UC OSS.
Higher-education call centers face costly, limited-coverage monitoring of advisor conversations and brittle, slow methods for identifying student concerns from transcripts. The proposed workflow deploys OpenAI Whisper on Databricks Model Serving, applies AI Functions for sentiment, topics, intent, and rubric scoring, and uses Unity Catalog to govern the resulting data. For advisor quality, an LLM-as-a-judge evaluates every transcript against a reference-table rubric, returns a weighted 1–5 overall score and per-criterion scores, and routes flagged calls for targeted QA review instead of random sampling. For student insights, quarterly transcript enrichment feeds an Agent Bricks Knowledge Assistant for cited reasoning over raw calls and a Genie Space for structured trend queries, while LangGraph orchestrates UC SQL functions as tools. Together, these components let non-technical advisors, mentors, and QA managers query student interactions without reaching out to a data SME.
Enterprise AI pilots often move data into separate vector databases, SaaS LLMs, or serving layers, creating governance gaps, compounded latency, fragmented costs and observability, and duplicated lifecycle work. The post advocates data-native agents: models, agents, tools, retrieval, and memory run inside the governed data platform, with policy enforced during query planning and computation rather than after responses are produced. It argues that post-hoc controls cannot undo sensitive information encoded in aggregations and can trigger token-burning retry loops. For state and memory, it presents Lakebase, managed PostgreSQL within Databricks, as transactional storage and a shared source of truth for multi-agent swarms. The described platform pattern combines Unity Catalog, Unity AI Gateway, Model Serving, MLflow 3, AI Search, Lakebase, and business-context services, and recommends inventorying workloads already outside the perimeter before closing seams incrementally.
Databricks presents Lakebase as a fully managed, serverless, standard Postgres database for combining operational and analytical workloads on its Data + AI Platform. The platform separates compute from storage, integrates with the lakehouse through Synced Tables and Lakebase CDF, and uses Unity Catalog for governance; copy-on-write branching and autoscaling to zero are described as core infrastructure primitives. The post showcases partner-built, ready-to-deploy accelerators spanning technology, finance, marketing, sales, supply chain, human resources, customer service, and operations. Examples include PostgreSQL migration assessment, multi-agent Genie orchestration, stateful enterprise agents, autonomous data reliability, governed contact-center intelligence, and project operations management. These offerings package Lakebase patterns into migration controls, domain-specific solutions, and agent frameworks intended to accelerate modernization and reduce transformation complexity.
Python app hosting is presented as both an infrastructure choice and, for data-intensive or AI applications, a data-architecture choice. The guide defines a hosting stack comprising application code, runtime, dependency manager, application server and reverse proxy, and contrasts it with traditional web hosting. It compares shared hosting, VPSs and cloud VMs, PaaS, container platforms and serverless functions by control, operational effort, scalability and workload fit. It notes that long-running data and AI tasks such as model training, vector indexes, cached datasets, ETL jobs and lengthy inference pipelines can exceed practical serverless limits. The conclusion is that public APIs and lightweight web apps allow broad platform choice, while apps accessing governed data, model endpoints or AI agents should be evaluated by where data lives and how access is governed, with Databricks Apps presented as an option when data already resides in a lakehouse.
Feature Views are presented as a framework for creating managed feature pipelines in Databricks, addressing duplicated logic, training/serving skew, backfills, online-store plumbing, and governance for real-time ML. A Feature View defines the source, entity, time-series column, and computation once; Databricks uses it to generate point-in-time-accurate historical data, then materializes pipelines for batch or streaming production use. Registered in Unity Catalog, materialized features receive lineage and access controls, while MLflow records model dependencies and Model Serving retrieves required features without custom lookup code. For Kafka-sourced streaming features, Spark Realtime Mode, Lakebase, and Model Serving support rolling-window updates with reported end-to-end p99 latency of 200ms from event to online availability. The announcement says Feature Views are in Public Preview and can be developed through the Feature Engineering Client SDK and Genie Code, with streaming materialization requiring an Enterprise-tier workspace in a region supporting Lakebase.
Health systems face an ambulatory intelligence gap: access, provider capacity, referral retention, panel management, and financial performance are interconnected, while information remains scattered across disconnected systems. Health Catalyst’s Ambulatory Intelligence combines AI with nearly two decades of healthcare improvement expertise, deploying directly in a customer’s Databricks workspace so sensitive data stays within the health system’s environment. It uses a medallion-based semantic layer, Unity Catalog for governance, Lakebase for low-latency serving, and Genie alongside dashboards to help leaders investigate why metrics change. The solution ships with prebuilt metrics across Access Optimization, Revenue Intelligence, Panel Management, and Referral Insights, plus cross-domain scorecards and configurable terminology and workflows. Reported outcomes from supported improvement work include increased revenue and encounters at Thibodaux Regional, 55,000 closed care gaps at INTEGRIS Health, and higher outpatient visits with fewer cancellations without reschedules at WakeMed; future plans include models based on prior outcomes and agentic capabilities.
The fifth Databricks hackathon used Databricks Genie to test three ways of working with governed data: asking, building, and composing. Genie Agents give business users domain-specific natural-language access to curated data, while Genie Code helps analysts create metric views, functions, pipelines, and dashboards inside Databricks. The composition track uses Conversation APIs and a managed MCP server to make Genie an agent tool alongside custom logic, Model Serving, and other MCP servers, with Unity Catalog governing access. Projects included supervisory routing across 190 tables, governance and analytics products built in days, and multi-agent systems such as ShipBob’s overnight operations brief. Across the tracks, the stated lesson is that shared governance and semantic context let business users, builders, and engineers use Genie at different levels without abandoning grounded, permissioned data access.
The best engineering reads, once a week.
