Loading…
Topic
137 posts about Unity Catalog. Every summary links to the original.
See the idea
An agentic ad stack wires governed customer signals to campaign execution through a shared tool interface and server-side guardrails. The diagram shows each layer, the direction of a tool call, and where control stays human.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Databricks made multi-statement transactions generally available on Delta Lake and Apache Iceberg tables, providing atomic guarantees for complex, multi-step workflows directly on the lakehouse. Previously, organizations had to rely on separate warehouse systems for these transactional guarantees, adding expensive data movement, duplicate copies, fragmented governance, and custom consistency workarounds. Row-level concurrency enables concurrent transactions to write to the same table as long as they do not update the same row, delivering significantly stronger performance than traditional cloud and legacy warehouses. The transactions are composable with stored procedures, temporary tables, and streaming tables, and they respect Unity Catalog attribute-based access controls including row-level security and column masking on both reads and writes. Fivetran used transactions to remove consistency-repair code for its slowly changing dimension updates using SCD Type 2, and Accolade processes roughly 1,700 transactions daily for claims and liabilities on the governed lakehouse.
Lakeflow Designer, a visual no-code data preparation tool in Databricks, added three new operators for cleaning, deduplication, and quality checks, addressing customer requests for more no-code capabilities. The Prepare operator standardizes data by changing case, converting types, replacing values, and applying formulas; Unique removes duplicates across selected columns with ordering control; Guardrails enforces quality rules with configurable failure handling. Existing operators gained multi-table joins with separate outputs for matched, left unmatched, and right unmatched rows, branching in Filter, and dynamic column selection in Select. Output options expanded to materialized views in Unity Catalog that refresh automatically, merge into existing tables, and direct file writes to CSV, Excel, or JSON with dynamic worksheets. Canvas improvements include a redesigned interface with better contrast and larger connection targets for large workflows, a searchable table of contents, visual version history, and live cursor presence.
Databricks announced the public beta of a Workday Data Connect federation connector for Unity Catalog, enabling zero-copy queries of Workday HR and finance data directly from the Databricks Data and AI Platform without ingestion pipelines. Workday shares tables via an Iceberg REST catalog; Databricks creates a foreign catalog in Unity Catalog and reads data from Workday’s managed object storage, with Unity Catalog governing read-only access through table-level permissions, lineage, and auditing. Teams can query the federated data using Databricks SQL, AI/BI, notebooks, or Genie for natural-language questions about workforce and financial trends. The connector requires Databricks Runtime 19 or above and a workspace admin to enable it from the Previews page. The release builds on the Databricks-Workday partnership announced at Workday Rising 2025 and complements existing Lakeflow Connect and Live Data Query options.
Databricks Apps now generally supports on-behalf-of-user (OBO) authorization, enabling apps to act using the signed-in user’s identity so Unity Catalog enforces existing data permissions including row filters and column masks. Developers choose between app authorization for app-owned operations and user authorization for user-specific data access, with API scopes such as sql:restricted-query limiting what the app can do on the user’s behalf. The forwarded user access token is short-lived and per-request, passed via the x-forwarded-access-token HTTP header to the SQL connector, and the app must never store it. Workspace administrators can set an allowlist of API scopes, providing two levels of control: the app declares minimum scopes and the workspace defines the maximum, with account administrators able to add scopes outside the allowlist. The post recommends keeping app-owned and user-authorized clients separate in code and failing closed when a user token is missing to avoid returning results with incorrect permissions.
General-purpose AI agents fail on business questions because they lack enterprise context such as authoritative tables, business rules, and trusted experts; the missing institutional knowledge is scattered across dashboards, documents, and people. Databricks’ Genie Ontology addresses this by continuously updating institutional knowledge from certified Unity Catalog concepts and automatically learned snippets from dashboards, notebooks, SQL queries, and connected apps. OntoRank ranks sources by authority using certification, usage, and authorship signals, so Genie grounds answers in the most trusted and relevant data. In a weekly adoption review, a product manager prompts Genie, which uses MCP to search Google Docs and Jira while running live SQL for fresh analysis. The resulting report includes citations, forecasting, scheduling, and even conversion into a shareable Genie Agent, all within permission boundaries.
Unity Catalog managed tables let you control where your data physically resides while keeping it in cloud storage you own, such as S3, ADLS, or GCS, unlike platforms that store tables in provider-controlled storage. You set managed storage locations at the metastore, catalog, or schema level, with the most specific level winning, and can change where new tables land using ALTER CATALOG or ALTER SCHEMA commands. Managed tables use open formats including Iceberg and Delta, and external engines such as Apache Spark, Flink, and Trino can read and write through Iceberg REST Catalog and Unity Catalog open APIs. When converting an external table to managed, Databricks copies the data and transaction log into the managed storage location you have set. This approach enables a single source of truth across analytics and AI workloads while satisfying compliance, cost allocation, and data residency requirements.
Genie One is an AI coworker that lets business users ask questions in plain language and receive cited answers grounded in governed data. The post outlines a phased rollout playbook: Phase 0 foundations include choosing one team, defining metric views for 10-20 numbers, creating Pages for concepts, and establishing governance with Unity Catalog. Phase 1 runs a 4-6 week pilot with 5-10 users, scoring Genie against 25-50 real questions and logging every incorrect answer to fix root causes, with exit criteria including accuracy holding steady and users reaching for Genie over their old workflow. Phase 2 expands to 2-3 adjacent domains with shared ontology definitions and a lightweight review workflow. Phase 3 aims for organization-wide adoption with federated ownership, MCP server governance, and embedding Genie in Slack, Teams, Excel, or Google Sheets, with a realistic timeline of 1-2 weeks for foundations to 1-2 quarters for full rollout.
Security reviews at Databricks relied on automation but still consumed expert time on predictable work, leaving less room for genuinely novel, high-risk architectures. The author built an agent-based layer on Databricks using Unity Catalog for governed tables containing standards, request data, and decisions; Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning; and Lakeflow Jobs for orchestrating notebook-based workflows on serverless compute. A conversational intake app built with Databricks Apps turns plain-language descriptions into structured requests and provides preliminary risk indications before a formal request is created. Seven focused agents each have a bounded responsibility—intake, risk assessment, requirements, specialized reviews, validation, workflow, and learning—ensuring behavior remains inspectable and testable. Routine requests that once waited days in the queue can now be completed in minutes, and security engineers spend more time on designs requiring genuine expertise.
Business concepts like active customer or completed trip are often defined differently across teams, scattered in Slack threads and spreadsheets, causing AI agents to guess confidently when answering questions. Databricks introduces Unity Catalog Pages as a governed home within Genie Ontology where data stewards with Genie Code define the authoritative meaning of those terms and entities. Pages complement existing semantics — metric views how to calculate, domains where to look, and certification what to trust — by capturing what each concept means. Genie Code can bootstrap Pages by extracting definitions from Confluence, Slack, Google Docs, or GitHub via MCP-connected tools, turning weeks of work into minutes. When Genie One uses a Page to ground an answer, the Page is cited as a clickable source so users can inspect the definition and verify the reasoning.
Databricks says it was recognized as a Customers’ Choice in Gartner Peer Insights’ Voice of the Customer for Analytics and Business Intelligence Platforms, based on verified reviews from customers using platforms in production. The post reports a 4.8 out of 5 overall rating and 94% willingness to recommend, while noting that the designation reflects customer adoption and overall experience meeting or exceeding market averages. It presents Databricks AI/BI, AI/BI Dashboards, Genie Spaces, and Genie as a unified architecture combining governed live data, AI, business intelligence, and Unity Catalog controls without moving or duplicating datasets. The described use cases include conversational exploration, dashboard and visualization development, consolidating fragmented reporting, and expanding governed analytics access, with the company positioning these capabilities as a foundation for agentic analytics.
HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.
Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.
Databricks announces the general availability of Unity Catalog Business Semantics, a unified semantic foundation for consistent business definitions across BI dashboards, notebooks, SQL applications, and AI agents. Built into Unity Catalog, it applies existing governance, security, lineage, and access controls to centrally defined Metric Views, whose metadata includes display names, formats, and synonyms. Metric Views are authored declaratively in SQL, compiled and executed deterministically at query time, and can use materializations with automatic pre-aggregation, incremental refresh, intelligent query rewriting, and transparent routing; materialization remains in Preview. The core Metric View implementation is being open sourced in Apache Spark OSS for an upcoming release, with Unity Catalog OSS v0.5 support coming soon, while a point-and-click authoring UI is in Public Preview.
Addepar, a global technology and data platform, manages nearly $9 trillion in assets and moved from older systems to a unified data intelligence platform on Databricks running on AWS. It ingests hundreds of disparate data feeds, standardizes and enriches them, and delivers results through products, APIs and data sharing. Since adoption, pipeline costs fell 60% versus legacy infrastructure, saving more than $2 million, while new pipelines and integrations became five times faster. Addison, a native AI experience embedded in the platform, combines governed portfolio Q&A, proactive insights and action-oriented workflows, using live data and Addepar’s core calculations engine. Addepar is adopting Databricks Agent Bricks, including Supervisor Agent flows with Genie-powered analytics, while Foundation Model Serving and managed MLflow support governed model serving, traceability, evaluation and iteration for human-supervised agentic workflows.
An agent that keeps a company’s market view current must combine Databricks-hosted internal account records with external web signals such as funding rounds, leadership changes, product launches, and hiring surges. The initial implementation duplicates enrichment logic across Claude Code, Codex, and a direct model API, with inconsistent search results and no shared cost, governance, or audit trail. Omnigent consolidates the agent definition, model, tools, policies, limits, and model-call governance above those harnesses, while letting the engineer select a common search provider. Nimble’s Search API supplies live web data, while its Web Search Agents orchestrate multi-source retrieval and extraction, cross-check sources, return citations, and adapt to difficult sites and recurring retrieval paths. In Nimble’s testing, adding its search increased LLM benchmark accuracy from 46% to 71% and halved web search costs; the proposed Databricks setup then connects internal records and external context under one governance surface.
Energy theft detection often stops at an ML risk score, leaving revenue-protection teams to manually interpret cases, prepare investigations, dispatch crews, recover losses, and report results. The described Databricks workflow turns that score into a governed operational loop: Lakeflow prepares features, models run through Model Serving and Unity Gateway, and a Databricks App uses Lakebase to store live case state. AI-generated summaries explain why accounts were flagged, while dispatch-ready reports provide next steps, evidence checklists, and safety notes. Genie One answers leadership questions using metric definitions in Unity Catalog, and an Agent Bricks Multi-Agent Supervisor can assemble a traceable month-end board pack. Unity Catalog lineage, PII labels, access controls, model guardrails, and Unity Gateway routing support regulated use under frameworks including Ofgem’s and ELEXON’s requirements, while the same pattern is presented as applicable to other ML-driven interventions.
Databricks marketers struggled to get trusted answers from fragmented marketing data quickly, prompting the creation of Marge, a marketing implementation of Genie Agents. The team first unified campaign, sales, web, event and advertising information in a governed Marketing Lakehouse, then grounded the assistant in business definitions, verified logic and example queries. Marge translates natural-language questions into analytical queries, uses Unity Catalog for lineage and role-based access, and can ask clarifying questions or perform multi-step analysis. The reported results include threefold greater data use in decisions, more than 85% adoption, 50% quarter-over-quarter growth, over 5,000 answered questions and a 25% reduction in flagged incorrect responses. Databricks attributes the rollout to continuous feedback, benchmark evaluation, focused use cases and embedding Genie in existing workflows rather than treating conversational analytics as a one-time deployment.
Lakeflow Connect addresses the fragmentation of Customer 360 data across ad platforms, analytics tools, CRMs, and support systems. Its fully managed native connectors ingest sources through a UI or API, incrementally landing governed tables in Unity Catalog without infrastructure to provision, manage, or scale. Connectors for Google Ads, Meta Ads, TikTok Ads, Reddit Ads, and LinkedIn Ads provide pre-built or custom reports with spend, impressions, clicks, and conversions, while OAuth flows simplify setup for some platforms. The post also describes HubSpot, Salesforce, and other journey-wide connectors, plus the Ad-Genie accelerator, which transforms bronze data into silver and gold tables for AI/BI dashboards and Genie agents. Customer examples report reduced engineering overhead and faster ingestion.
Embedding a Databricks AI/BI Dashboard in a customer-facing application is straightforward; authorization determines which rows each viewer may see. The pattern supports external partners and internal Databricks teams with one dashboard and shared dataset, while applying different regional access and masking rules. An entitlements table stores partner IDs and IdP-synchronized group names alongside regions and PII-mask flags; a secured view filters on the signed __aibi_external_value from the embed token. Direct SQL access is separately protected by Unity Catalog row filters and column masks, which evaluate the querying user's identity and groups. The design provides a default-deny, pooled logical multi-tenancy model that centralizes rules while covering both application and workspace query paths.
Long-running personal-loan underwriting agents must preserve completed work, resume after worker failures, tolerate repeated tool calls, wait for reviewers, expose operational state, and retain an audit trail. The reference implementation combines Temporal durable Workflows and Activities with Lakebase Postgres, using Temporal Event History for replay and Lakebase schemas for run status, evidence, review state, messages, and metrics. Deterministic identifiers, constraints, guarded updates, and Postgres upserts make repeated Activity attempts target the same logical records, while Signals carry human decisions and long waits do not occupy workers. A Unity Catalog synced table supplies governed underwriting policy, and optional Change Data Feed publishes operational changes to Delta history tables. The implementation includes a React/FastAPI interface, mocked deterministic providers, OAuth credential refresh, and 21 passing tests, but does not validate production lending, compliance, security, scale, or an observed end-to-end feed.
The best engineering reads, once a week.
