Loading…
Topic
136 posts about Unity Catalog. Every summary links to the original.
See the idea
An agentic ad stack wires governed customer signals to campaign execution through a shared tool interface and server-side guardrails. The diagram shows each layer, the direction of a tool call, and where control stays human.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Financial services organizations have built sophisticated lakehouses, streaming pipelines, model-serving infrastructure, and self-service BI, but access remains concentrated among technical teams. Business leaders still often rely on analysts because they may lack SQL skills, BI training, or analyst access, creating the “last mile” of data democratization. Databricks Genie addresses this gap through a conversational AI interface that converts plain-English questions into governed SQL queries executed against the Databricks Lakehouse without an analyst in the loop. It operates within Unity Catalog access policies, restricts users to authorized data, makes queries read-only, and logs interactions for audit purposes, while its semantic layer maps organizational terminology such as NIM, LTV, and NII to the organization’s meanings. The stated outcome is faster, auditable answers for business questions and usage data that can inform data-product priorities.
Databricks presents a first cohort of partner-built industry solutions that use Genie to reduce the “Analyst Bottleneck,” where leaders wait for custom SQL or dashboard updates. Genie provides a conversational analytics layer grounded in Unity Catalog metadata and business semantics, using specialized AI agents to return governed, secure, traceable answers to natural-language questions. The featured accelerators span communications, media and entertainment, financial services, healthcare and life sciences, manufacturing and energy, public sector, and retail, with examples including churn prediction, next-best-action recommendations, billing anomaly investigation, KYC monitoring, actuarial analysis, and commercial insights. Across the examples, solutions combine domain expertise with Databricks platform capabilities such as machine learning, Agent Bricks, Lakebase, Serverless SQL, and auditable governance to support self-service analysis and operational decisions.
Media companies may have launched streaming services, digital editions, and mobile apps, yet still face a digital product intelligence gap: product teams must answer behavioral data questions quickly enough to personalize experiences and improve engagement. Databricks Genie gives nontechnical product leaders a conversational interface that translates natural-language questions into SQL queries, visualizations, and actionable insights over governed enterprise data, without requiring code or analyst handoffs. The agent queries governed Delta Lake tables fed by streaming clickstreams, views, and session signals, combining event-level behavior, A/B tests, audience segments, and cross-platform data in conversational answers. The source says internal benchmark accuracy improved from 32% to over 90% through multi-LLM orchestration, specialized knowledge search, and parallel reasoning, and presents Genie as reliable enough for production personalization decisions.
Databricks Genie makes descriptive analytics accessible in natural language, but predictive questions still require specialized data science workflows and carefully prepared datasets. This post presents a multi-agent supervisor deployed as a Databricks App, combining Genie, TabPFN, and Agent Bricks to turn business questions into predictions. The orchestrator asks Genie to use governed Lakehouse data, schemas, relationships, and semantics to generate labeled training data through SQL, then sends it to TabPFN, which predicts in a single forward pass without feature preprocessing, model selection, or hyperparameter tuning. The resulting conversational experience supports descriptive and predictive analytics with Unity Catalog lineage and access control, while an MLflow GenAI evaluation harness monitors reliability and regressions. Its central limitation is that predictions depend on Genie producing a meaningful dataset with a clear label, so missing signals, joins, outcomes, or agent omissions can make results unreliable.
With AI agents proliferating across organizational functions, governance must answer which agents access sensitive data and how they act. Traditional approaches either scatter logs and permissions across systems or restrict deployment, while agents require controls over autonomous tool use and runtime behavior. Unity Catalog and Unity AI Gateway address this through four pillars: delegated access, data-centric governance, cost intelligence, and open interoperability. The design passes user identity through agents with on-behalf-of tokens, evaluates MCP tool calls with Service Policies, applies inline guardrails, and records model, access, trace, and usage data in lakehouse tables. It also connects data quality, classification, spend, and outcomes, while supporting multiple frameworks and model providers through shared governance.
Retail personalization often stalls when CX leaders must wait for analysts to answer questions about segment behavior, loyalty performance, channel preference, or churn risk, narrowing the window for action. Databricks Genie addresses this access gap by letting business users query unified structured and unstructured enterprise data in plain English instead of SQL. Its retail differentiators include identity-resolved queries across channels and devices, lifecycle-stage awareness, campaign response and control-group data integration, and privacy controls enforced through Unity Catalog. The source says Genie reduces routine analyst requests rather than replacing data science teams, while enabling merchandisers, category managers, loyalty marketers, and CX leaders to self-serve operational questions. It cites 7-Eleven’s use of Databricks SQL, Unity Catalog, and AI/BI Genie to launch, refine, and measure personalized offers within a secure, unified platform.
Agentic AI systems connected to external tools can take destructive production actions while operating within delegated authority, yet without restrictions on tool selection or records of their calls. Unity Catalog now governs MCP tools with fine-grained access control, service policies, and payload logging, while Unity AI Gateway enforces decisions in real time. Policies are SQL functions receiving actor and context information; they can allow, deny, or require user consent, inspect arguments and caller properties, and block calls before execution. Every tool call is recorded in a Unity Catalog-managed Delta table with the tool name, arguments, result, user identity, and allow-or-deny status, making the activity queryable with SQL. The service policies and payload logging are available as a Gated Beta, with early access offered through Databricks account teams.
Databricks announces the Databricks Certified Context Engineer Associate, described as the industry’s first certification purpose-built for context-aware AI systems. It assesses whether practitioners can design, assemble, and govern the information AI agents receive at inference time, including system prompts, retrieval, memory, external tools, context-window management, and governance. Candidates work with Databricks technologies such as AI Search, Lakebase, MLflow, MCP, and Unity Catalog, while advanced coverage includes multi-agent systems, long-horizon workflows, and evaluation of context decisions. The beta exam will be available free to Data + AI Summit attendees, who may take it once; beta results are expected in 6–8 weeks.
In ATTR-CM, a progressive and often fatal form of heart failure, 70 to 80 percent of patients do not know they have the disease, creating a need for better commercial prioritization than static call lists provide. Databricks and Veeva position embedded Genie agents and AI/BI dashboards in Veeva Vault CRM as a way to connect commercial data bidirectionally and answer role-specific questions inside existing workflows. Sales reps can see geographic HCP views with suspected patient counts, formulary scores, office availability, NRx-weighted priorities, and talking points, then rebuild a day after a cancellation. MSLs can generate cited pre-call briefs from approved sources such as Veeva Link, PubMed, ClinicalTrials.gov, and ASNC guidelines. Territory managers receive personalized views of call patterns, unworked signals, and dormant HCPs, while Unity Catalog provides shared access, lineage, and compliance governance.
Part 2 shows how moving Backstage’s operational Postgres database to Databricks Lakebase brings security, auditing, masking, and cost controls under Unity Catalog. The proof of concept used Lakehouse Federation to expose the catalog as the lakebase_bs foreign catalog, replacing cross-service RDS investigations with Unity Catalog grants and system.access.audit records. Audit events identify OAuth users and source IPs, while billing attributed 31.6130 DBU to production and 0.0107 DBU to a dropped test branch. Branch-level attribute masking propagates automatically to feature, CI, and QA copies, while the post cites a Perforce report stating that 60% of organizations experienced breaches or theft in non-production environments. It also introduces LakebaseOps, with three agents and seven scheduled jobs, and Lakebase MCP, exposing 46 governed tools across four profiles, so DBAs can design policies and workflows instead of handling repetitive provisioning.
Unity Catalog’s expanded Open APIs address data silos by letting organizations use multiple compute engines while retaining centralized governance and a single copy of data. In beta, Apache Spark, Apache Flink, and DuckDB can create, read, write, and stream to or from UC managed Delta tables, with catalog commits providing serialized commits, transactional safety, and auditability. Delta Kernel, an open source Java and Rust library, abstracts low-level protocol details, helping connectors integrate external writes with catalog-managed commits while Predictive Optimization continues to run on accessed tables. Credential vending, now GA for tables, issues short-lived, scoped cloud credentials and supports M2M OAuth plus automatic refresh; volume credential vending is in Public Preview for unstructured data. The roadmap includes functionality for fine-grained row- and column-level ABAC on external reads, while external managed-table access remains in beta.
NYU Langone Health’s AI strategy starts with data quality, arguing that healthcare AI cannot be reliable when source data is fragmented or inconsistent. The institution standardized on common transactional platforms, including one electronic health record and one ERP system, established authoritative data sources, and fixes data at the source rather than mapping it in the warehouse layer. Its Databricks-based unified data and AI platform, with Unity Catalog, supports clinicians, analysts, scientists, and corporate users across care, operations, and research, while real-time feeds power emergency-room decision-support models. Mherabi also describes a three-layer analytics model: structured visualizations, conversational tools such as Genie, and answers delivered in formats suited to the user. The stated conclusion is that upstream data discipline, governance, literacy, and adaptable platforms provide the foundation for trustworthy AI and timely clinical insight.
Unity Catalog now generally offers three complementary data-governance capabilities: Attribute-Based Access Control (ABAC) policies for row filtering and column masking, Governed Tags, and agentic Data Classification. They address per-object access rules, coordination gaps, and manual detection by letting governance teams define tag-based policies once, automatically classify sensitive data, and protect matching objects across catalogs and schemas. Governed tags provide an account-level vocabulary inherited across catalogs, schemas, tables, and columns, while ABAC applies row filters and column masks using tag-based conditions. Classification uses built-in compliance classifiers, custom classifiers, metadata, pattern recognition, and large language models, with human-in-the-loop validation and false-positive exclusions. General availability adds 10x larger policy limits, support for 10,000+ policies per metastore, lifecycle management through SQL, APIs, UI, and Terraform, expanded compliance coverage, and custom classifiers in beta.
Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis.
MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.
State, tribal, local, and territorial (STLT) health agencies manage data across surveillance, vital records, Medicaid, WIC, and emergency preparedness systems, but those systems are fragmented and difficult to query together. That limits rapid public health intelligence: questions linking emergency-department visits with pharmacy dispensing, school absenteeism, vaccination, demographic, or geographic data can require epidemiologists to assemble manual queries over weeks, even when decisions require answers within hours. The post presents Databricks Genie as a natural-language interface for querying this environment, backed by a Databricks engine that handles petabyte-scale datasets across real-time streams and historical records. It describes cross-program synthesis, Unity Catalog row- and column-level access controls, HIPAA-compliant governance, traceability to the underlying query, and validation controlled by health experts. Examples include county-level influenza-like illness trends overlaid with vaccination coverage and identifying counties with high opioid overdose rates and low treatment utilization; Genie is described as available today.
Federal agencies have invested substantially in data infrastructure, but program directors, policy analysts, oversight officials, and budget examiners often still rely on technical intermediaries to answer operational questions. The post frames this as the unresolved “last mile” of federal data modernization: data lakes, APIs, dashboards, evidence-based policymaking mandates, and agency CDO functions have advanced infrastructure without making it usable by most decision-makers. It presents Databricks Genie as a natural-language interface that lets staff query agency data in plain language, including questions requiring joins across disbursement, eligibility, and geographic data, while retaining existing access controls and policies. Genie runs on Unity Catalog with role-based access controls, audit logging, and data lineage; it also supports federated cross-agency queries and records queries, answers, and sources, which the post associates with oversight, FOIA readiness, and accountability.
Financial institutions have invested heavily in model governance frameworks, stress testing infrastructure, limit monitoring, data feeds, and dashboards, but risk leaders may still lack fast access to what those models are telling them. When a CRO must assess credit concentration, scenario sensitivity, or relationships between market positions and credit exposures, answering can require navigation across model outputs, analyst interpretation, and disconnected data systems. The proposed approach uses Databricks AI/BI Genie to let leaders query risk data in natural language, while retaining access controls, audit logging, Unity Catalog lineage, cross-risk data, and stress-test outputs in one environment. The stated distinction is that governance establishes necessary controls, whereas conversational risk intelligence supports questions that fixed dashboards did not anticipate, including comparisons with internal limits.
SAP Business Data Cloud now offers generally available semantic metadata synchronization with Databricks Unity Catalog for mounted SAP BDC Delta Shares. When a table is accessed, Unity Catalog receives table-level business-friendly display names, descriptions, context, and governance tags, while SAP BDC remains the single source of truth and subsequent changes are reflected. The capability builds on BDC Connect and Delta Sharing, allowing governed SAP data products to be discovered and combined with other enterprise sources without recreating business context or governance separately. Column descriptions, table relationships such as primary and foreign keys, and PersonalData namespace tags give Databricks AI Assistant and AI/BI Genie explicit context for natural-language questions and join-ready queries. The stated result is more understandable, discoverable, and AI-ready SAP data, with automated classification signals supporting compliance, access control, and responsible AI.
Stripe Data Pipeline is now available on Databricks Marketplace through Delta Sharing, allowing payment and business data to be analyzed in a Databricks workspace without ETL. Stripe data—including transaction records, customer histories, subscriptions, refunds, and payouts—remains in Stripe’s infrastructure and is queried directly through Unity Catalog, where it can be joined with other tables. The approach replaces polling integrations and custom jobs, avoiding per-call charges, connector licensing fees, data duplication, and maintenance code while providing row- and column-level access controls, audit trails, and compliance features. The shared data is positioned for AI-native uses such as continuous transaction monitoring, churn scoring, LLM-powered retention workflows, Genie queries, and finance analytics apps. Getting started involves visiting the Stripe Data Pipeline listing on Databricks Marketplace.
The best engineering reads, once a week.
