Loading…
Topic
136 posts about Unity Catalog. Every summary links to the original.
See the idea
An agentic ad stack wires governed customer signals to campaign execution through a shared tool interface and server-side guardrails. The diagram shows each layer, the direction of a tool call, and where control stays human.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Traditional customer data platforms (CDPs) were built for human-managed, batch-based campaigns, but the post argues that agentic buying requires millisecond speed, hyper-personalization, and richer context. It contrasts the familiar Golden Record with Golden Context, which combines customer data with current business goals and the history and outcomes of prior decisions. The proposed Agentic CDP uses “Infinity Campaigns,” always-on engagement loops that use LLMs and agents to adapt messaging, timing, and channels for individuals. It is also embedded in the data foundation, bringing customer, business, and decision context together under existing governance, and is designed for agents and humans from the outset. Databricks presents CustomerLake as an implementation of these principles for its platform.
Databricks introduces three capabilities intended to bring vibe coding to enterprise applications, where speed alone does not provide business-data context, deployment safety, or cost control. App Spaces lets admins define resource and data access, on-behalf-of-user API scopes, and security policies for groups of apps, with each app inheriting those settings. Genie App Builder turns plain-language descriptions into working internal apps through generated plans, live previews, AppKit, and awareness of workspace data assets and Unity Catalog semantics. Serverless micro apps run in isolated lightweight virtual machines, start quickly when needed, scale to zero while idle, and use usage-based rather than reserved-capacity infrastructure. Together, the capabilities are presented as a way for business-proximate users to build on enterprise data while organizations apply consistent governance and support broader app portfolios; all three are coming to Databricks Apps, with private previews coming soon.
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
Databricks announces the Public Preview of Apps on Databricks Marketplace, allowing customers to discover, install, and run third-party data and AI applications inside secure Databricks workspaces. The offering addresses procurement challenges involving data movement, lengthy security reviews, custom integrations, and fragmented identity management by bringing applications to the customer’s data. Installed apps run in isolated sandboxes within the consumer’s Databricks account, inherit Unity Catalog governance, and use dedicated serverless compute with consumer-controlled external access through Serverless Egress Gateway policies. Providers can publish closed-source containerized apps once for no-egress distribution without maintaining per-customer infrastructure, while applications connect natively to services including SQL Warehouse, Lakebase, Model Serving, and Foundation Model APIs. The Public Preview launches with 20 partners, and planned additions include bundled assets, provider analytics, and commercial monetization.
At Data + AI Summit 2026, Unity Catalog announcements position the catalog as a runtime governance layer for enterprise data and AI, organized around control, context, and choice. Control additions include Unity AI Gateway for governing models, agents, MCP services, skills, and tools; contextual service policies can allow, deny, or require approval for runtime actions, while budgets, hard caps, tracing, and guardrails address spend, investigation, and safety. Context additions include Glossary and Domains for business meaning and scoped asset organization, plus Metrics that standardize KPIs for SQL, BI tools, APIs, and agents; Genie Ontology is described as a continuously learned enterprise context layer. Choice additions span cross-cloud and cross-region addressability, managed disaster recovery, Delta and Iceberg interoperability, multimodal and geospatial types, and open sharing of data, AI assets, and applications across organizations.
At Data + AI Summit 2026, Databricks announced Agent Bricks as a comprehensive developer platform for building and operating agents, extending a product launched the previous year. The announcement frames the core agent loop as only 1% of the work, with token capacity, deployment, security, evaluation, monitoring, context, and sharing forming the remaining infrastructure burden. Agent Bricks addresses choice, context, and control through support for multiple proprietary, open-source, and custom models, any agent harness, MCP-connected data, Genie Ontology, managed memory, document intelligence, sandboxes, and governed tools. Unity AI Gateway adds catalogs, fine-grained access controls, budgets, traffic routing, contextual policies, monitoring, and registry support for agents, tools, and models. Databricks says more than 100,000 agents have been built and customers including AstraZeneca, 7-Eleven, Fox Corporation, and Block have shipped agents on the platform.
Databricks announces new Unity AI Gateway capabilities for governing enterprise AI as organizations operate multi-model, multi-agent, and multi-vendor estates connected to models, MCP services, APIs, and tools. The update adds unified spend visibility, granular attribution, hard spend caps, and smart routing, alongside Unity Catalog support for registering and governing models, MCP services, agents, and skills. Contextual Service Policies, in Beta, can allow, deny, or require approval for actions based on users, agents, models, tools, services, or request and response contents, with guardrails for risks such as PII exposure and prompt injection. The announcement also covers end-to-end tracing, coding-agent analysis with Genie, incident investigation with Lakewatch, ecosystem integrations, and Managed Omnigent on Databricks in Beta.
Databricks announces a major evolution of Lakeflow, its unified platform for data engineering across ingestion, transformation, and orchestration, with capabilities centrally governed by Unity Catalog. Genie Code and generally available Lakeflow Designer support agentic and no-code pipeline development, while Genie ZeroOps monitors production assets, analyzes failures, proposes fixes, and validates them in a governed sandbox before human approval. Lakeflow Connect expands to more than 100 managed connectors, and Zerobus Ingest adds Kafka-compatible, gRPC, REST, SDK, and OpenTelemetry interfaces for high-volume event ingestion. Real-Time Mode for Spark Declarative Pipelines reaches Public Preview with end-to-end latency as low as 5 milliseconds, alongside declarative APIs and expanded Lakeflow Jobs integrations. The release also adds data-readiness triggers and external orchestration for systems including Snowflake, REST APIs, Slack, and PagerDuty.
Genie ZeroOps is an autonomous background agent for monitoring and operating data and AI assets, including jobs, pipelines, tables, and ML models. It continuously detects visible and silent failures, uses Unity Catalog lineage and platform observability to assess root causes, generates remediation through development workflows, and verifies fixes in isolated sandboxes. These environments use shallow, zero-copy table clones, scoped permissions, and network isolation, so proposed changes run against real data without touching production or applying anything before approval. For ML workloads, the agent can diagnose degraded predictions, train a candidate on corrected features, evaluate it against the production model’s existing eval suite and criteria, and support live-traffic ramping when it is measurably better. Genie ZeroOps is entering private preview in the coming weeks, initially supporting jobs, pipelines, tables, and ML workloads; Apps and Lakebase databases are on the roadmap.
At Data + AI Summit 2026, Azure Databricks announced capabilities aimed at moving enterprises from experimental AI pilots to production-grade automated workflows by unifying data, productivity tools, marketing, and governance on Azure. Its Agentic Data foundation introduces LTAP, combining analytical data, streaming pipelines, and live application transactions in one lakehouse storage copy; Lakebase adds a managed serverless Postgres engine with copy-on-write branching, while Lakehouse//RT targets millisecond responses for high-concurrency workloads. Genie integrations for Microsoft Teams, M365 Copilot, Excel, and SharePoint bring governed lakehouse intelligence and ingestion into daily work, alongside tools for agents, applications, pipelines, and autonomous operations. CustomerLake adds Profile Agents and Campaign Agents for customer profiles and personalization, while Genie Ontology and Unity AI Gateway provide semantic context, rate limits, content filtering, and spend controls.
Databricks announces Genie One, Genie Agents, and Genie Ontology to help enterprises answer business questions and act on data whose context is scattered across dashboards, queries, documents, tickets, and chats. Genie One connects data and business tools through Lakehouse federation, Lakeflow Connect, native integrations, Slack, Teams, mobile apps, schedules, alerts, document creation, custom skills, and MCP support. Genie Agents evolve Genie Spaces into domain-specific agents that can reason over structured and unstructured data, execute multi-step workflows, and be created from a prompt. Genie Ontology builds a permission-aware living graph from enterprise assets, weighting sources by authority, usage, certification, and freshness. In an internal 28-question benchmark, Genie answered 84.5% correctly on the first attempt and delivered twice the speed of the strongest coding agent.
Databricks announces CustomerLake, an Agentic Customer Data Platform embedded natively in its lakehouse, bringing Customer 360, identity resolution, audience building, campaign automation, activation, and personalization alongside governed data and AI models. The announcement addresses fragmented identities, stale audiences, manual campaign workflows, and the duplication and governance burden created by separate martech systems. CustomerLake uses Unity Catalog and Lakehouse Federation to access customer data across Databricks, Snowflake, Google BigQuery, cloud object storage, operational databases, and other enterprise systems, while Profile Agents create business-ready profiles and Campaign Agents build audiences, recommend actions, activate channels, and optimize engagement. Its operating model is described as embedded, democratized, and autonomous, with Agentic Identity Resolution combining deterministic, probabilistic, and agentic workflows. CustomerLake is now available in Private Preview and launches with an open partner ecosystem.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Databricks announces new OpenSharing and Marketplace capabilities aimed at sharing data, AI assets, and partner applications without moving data. OpenSharing, a Linux Foundation project and evolution of Delta Sharing, adds vendor-neutral sharing for Agent Skills, AI models, unstructured data, Iceberg clients, Lakebase tables and change data feed, plus governed multi-cloud connectivity and agentic sharing through Genie Agent Sharing. The release also introduces identity resolution in Databricks Clean Rooms, where partners such as LiveRamp and Acxiom can work against protected first-party data without seeing raw customer records. Third-party apps are now available through Databricks Marketplace, allowing customers to deploy them in their workspaces without separate infrastructure, while providers gain distribution. Marketplace Commit Drawdown lets customers use existing Databricks universal commits to acquire Marketplace data and AI solutions.
Document AI uses machine learning, natural language processing (NLP) and optical character recognition (OCR) to extract, classify and understand information from structured, semi-structured and unstructured documents. Unlike OCR alone, it interprets layout and context, turning files such as invoices, contracts and emails into structured, actionable data through ingestion, OCR, layout parsing, entity extraction, classification, validation and, when needed, human review. Modern systems add large language models for summarization, document Q&A and zero-shot extraction, but hallucination risk makes validation and human oversight essential, particularly in regulated settings. The guide also describes Databricks Document Intelligence, which processes and stores documents alongside organizational data under Unity Catalog, using AI Functions, Variant and Lakeflow Jobs to create governed, queryable workflows without moving data between systems.
Kim Hatton, Databricks’ Global Financial Services Marketing Leader, describes how two decades in regulated financial-services marketing led her toward technology and data-centered strategy. She says financial institutions still need to unlock value while navigating compliance, but their divisions and systems make a unified customer view difficult. Her account points to Unity Catalog’s unified governance and single source of truth for breaking down silos and supporting requirements including GDPR, customer identity, and sensitive workloads. It also describes Lakebase’s separation of compute and storage for faster ML/AI agent experimentation, alongside Genie’s plain-language data analysis, which can reduce work that otherwise takes months and expensive third parties. Hatton connects these tools with faster, more confident marketing and accurate decision-making in regulated workflows, while also describing Databricks’ inclusive, high-energy culture.
Enterprise intelligence (EI) is presented as an organization-wide capability combining business intelligence, knowledge management, enterprise search and AI to turn structured and unstructured data into decisions and actions. Unlike traditional BI, which centers on dashboards, reports and structured data, EI connects these capabilities through a shared architecture and governed business context. The described stack includes a lakehouse-based data foundation, batch and streaming pipelines, governance, semantics, analytics, search, machine learning, generative AI and a decision layer. Shared definitions such as “active customer” and “monthly revenue” are intended to keep dashboards, queries and AI agents aligned, while maintained context addresses knowledge that becomes stale as the business changes. The result described is a common trusted source from which people, applications and agents can produce insights and initiate actions.
The post concludes a series on how copy-on-write database branching in Databricks Lakebase changes team-scale evolutionary database development without changing its underlying methodology. For a team of fifty developers, long-running tier branches and ephemeral feature branches form a parent-linked promotion hierarchy, replacing separately provisioned environment instances and enabling promotion by merge, rollback by repoint, and computable schema divergence. Governance is declared once and inherited per branch, with policies intended to prevent transitions that contradict the parent chain; Unity Catalog captures metadata for attribution and audit. The DBA's role becomes platform engineering, while agents operate inside an executable SCM state machine with documented inputs, outputs, schema validation, and enforced gates. An optional TDD layer adds dedicated roles, acceptance-criterion scenarios, RED-GREEN-REFACTOR cycles, and artifact contracts, and the conclusion presents the resulting workflow as operational for human and agent practitioners.
Lakehouse Federation addresses the challenge of reasoning across enterprise data spread among AWS Glue, Snowflake, Oracle, BigQuery, Postgres, and legacy formats without first migrating it. By connecting external sources in place and syncing their metadata into Unity Catalog, Databricks applies shared permissions, lineage, and access controls while preserving source data. Federated comments and descriptions give Genie schema context, while Unity Catalog Semantics lets teams define governed metrics such as ROI once for consistent use across Genie, dashboards, and notebooks. The example connects an AWS Glue marketing database, carries its metadata, defines an ROI metric view, and asks Genie which campaigns led ROI last quarter. The post reports an immediate, accurate answer from live Glue data, and describes planned richer semantics, broader federation, and possible performance gains from managed tables.
Mercedes-Benz Korea piloted a “Talk to Data” architecture that extends its Databricks analytics foundation with a governed semantic layer for enterprise AI, rather than treating the effort as a chatbot project. The design moves Power BI DAX KPI logic into Unity Catalog Business Semantics and Metric Views, keeping sources, joins, measures, dimensions, comments, and synonyms alongside governed Lakehouse data. Genie spaces use curated metric views for domain questions, while Agent Bricks composes persona-based agents, with Unity Catalog enforcing row- and column-level access. An automated DAX-to-Metric-View transpiler parses semantic models, maps tables, generates draft definitions, flags non-automatable measures, and reports conversion gaps. The documented playbook combines gold-layer curation, KPI validation, regression testing, Genie optimization, persona agents, and Databricks Apps; the pilot reports AI answers aligned with established KPI definitions and BI reporting logic.
The best engineering reads, once a week.
