Loading…
Unity Catalog
112 posts about Unity Catalog. Every summary links to the original.
The next generation of Databricks Genie
The next generation of Databricks Genie expands the assistant beyond individual Genie Spaces, combining structured and unstructured enterprise data in a unified chat experience. It reuses logic from certified Genie Spaces, governed dashboards, and Databricks Apps, with metadata-based routing that prioritizes higher-trust sources, while new reasoning models and agent architecture handle questions spanning multiple domains. Built-in connectors for Google Drive and SharePoint, plus MCP support, let Genie access knowledge stores and take actions; Unity Catalog AI Gateway manages these connections. The experience replaces Databricks One as an account-level global home, adding domains, custom URLs, unified login, automated identity management, and governance through Unity Catalog. Native iOS and Android apps extend access to dashboards, apps, and chat beyond desktop environments.
Ken Wong, Dillon Morrison, Richard TomlinsonModel risk management in 2026: A banker's guide to the revised interagency guidance
The post presents the April 17, 2026 revision to model risk management guidance as a shift toward risk-based, principles-driven oversight, replacing several earlier issuances from the Federal Reserve, FDIC, and OCC. It identifies five operational changes: materiality tiering, end-to-end lifecycle governance, reproducible effective challenge, continuous drift and performance monitoring, and application of MRM principles to GenAI and agentic systems. Its proposed response is a Databricks reference architecture built on Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines, Feature Store, MLflow, Model Registry, Model Serving, and assurance tools. The design makes tiering metadata-driven and turns lineage, validation, approvals, monitoring, documentation, and retirement records into evidence generated during normal model work. The stated conclusion is that a unified substrate can reduce integration and manual evidence work, allowing regulatory changes to become configuration exercises rather than multi-quarter programs.
Pavithra Rao, Jennifer Miller, Chaitanya Varanasi, Kim HattonAnnouncing the Public Preview of Lakeflow Designer
Databricks announces the Public Preview of Lakeflow Designer, a visual, no-code, AI-native experience for data preparation and analytics built directly into the Databricks platform. It targets analysts, domain experts, and other less technical users with drag-and-drop workflows and natural-language authoring, while representing each transformation as an operator with previews. Genie Code can use Unity Catalog metadata, descriptions, lineage, popularity, and example queries to generate or modify workflows, inspect transformation results, and iterate when needed. Every visual transformation produces production-ready Python code that can be reviewed, versioned in Git, and scheduled through Lakeflow Jobs, while users pay for compute rather than per-user licenses. The preview is available in all workspaces, although an administrator may need to enable Visual data prep in the preview portal.
Jason Messer, Emanuel Zgraggen, V Maharajh, Matt Jones, Tracy YangHow to transform document activation workflows with Genie and Agent Bricks
Enterprise document workflows often require people to read PDFs, copy fields into spreadsheets, and re-enter data into business systems, creating delays, error risk, and weak auditability. The proposed alternative combines AI/BI Genie for natural-language queries over Delta tables, Agent Bricks for production agents, and Unity Catalog for permissions, lineage, and traceability. A five-phase workflow extracts fields into Lakeflow Bronze, Silver, and Gold layers, then supports structured queries, clause-level RAG answers, multi-agent routing, and actions through MCP servers. Examples include syncing validated rights data to ERP or CRM systems, updating billing entitlements, and triggering workflows from detected obligations. The approach targets media, advertising, ad tech, and telecommunications, where customers see improvements including faster month-end close, recovered revenue, reduced leakage, and lower operational risk.
Elena TesserBeyond the spreadsheet: How Databricks is delivering the modern CFO in financial services
Financial-services CFOs are being asked to act as Strategists and Catalysts, but fragmented legacy systems, T+1 batch processing, opaque reporting logic, and mismatched business semantics keep them focused on stewardship and retrospective operations. The proposed answer is Databricks as a unified, governed platform combining real-time streaming, centralized lineage, data and AI, with Unity Catalog, Lakeflow, Genie, and Agent Bricks addressing trust, latency, access, and model reproducibility. Unity Catalog can connect semantic definitions and end-to-end lineage from transactions through regulatory reports and models, while Lakeflow supports continuous ledger and liquidity processing. The post cites a global bank reducing liquidity-reporting processing from 10 hours to 8 minutes and Nationwide Insurance reporting a 5-point combined-ratio improvement and 3-point expense-ratio improvement. It presents a Lakehouse-based CFO stack as a shift from reporting historical results toward real-time capital management and previews AI-driven deposit and PPNR modeling.
Jennifer Miller, Marcela Granados, Andrea DeSosa, Alex Oberlander, Kim Hatton, Pavithra Rao, Naeem Rehman, Pravin Varma, Olga Deriy, Prasanna SelvarajMultimodal Data Integration: Production Architectures for Healthcare AI
Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.
Maks KhomutskyiIntroducing the Databricks Excel add-in for business users
Databricks has introduced the Excel Add-in, now in public preview, to let business users import and analyze lakehouse data directly in Excel without writing SQL or configuring ODBC drivers. Built on Databricks SQL and Unity Catalog, it provides point-and-click access to tables and metric views, native Excel pivot tables, refreshable results, reusable Workspace queries, and optional saved SQL. Metric views let data teams define business semantics once while Unity Catalog centrally manages permissions and controls, helping avoid duplicated logic and metric drift. The preview supports selecting tables and metric views, parameterized custom functions, manual refreshes, pivot-table filtering, query reuse, and workspace changes; scheduled refresh, AI integrations, and other usability improvements are planned next.
Toussaint Webb, Kaitlin BaumgardnerMercedes-Benz builds a cross-cloud data mesh with Delta Sharing and intelligent replication, cutting costs by 66%
Mercedes-Benz built a cross-cloud data mesh to share after-sales data between AWS and Azure while supporting its data-defined vehicle strategy. Existing direct queries incurred egress costs, and weekly full loads delayed updates for datasets that included vehicle over-the-air events and workshop visits; source tables were in Iceberg while many consumers expected Delta. Using Unity Catalog and Delta Sharing, the company federated AWS Glue tables, established cross-metastore shares, and used periodic serverless Sync Jobs with Delta Deep Clone to incrementally replicate selected tables into Azure object storage for local querying. This hybrid model preserves direct sharing for fresher workloads while reducing transfer for less time-sensitive ones. For the initial ten data products, egress costs fell 66%, update frequency improved from weekly to every second day, and the source reports an approximately 93% annual reduction in its 50-use-case comparison.
Alexander Summa, Aleksandar DragojevicIntroducing Genie Agent Mode
Databricks introduces Agent mode in Genie spaces, an agentic process that plans, explores, and reasons over data to answer business questions. For each question, Genie can confirm an issue, test potential contributors through multiple queries, reflect on results, and decide what to investigate next, using Unity Catalog metadata and author-defined semantics from the Genie space. It produces a findings report with quantified results, identified contributors, visualizations, and references to the underlying SQL, and may include actionable recommendations. The agent scales its reasoning to task complexity, using validation for straightforward questions and more planning for multi-step investigations; in AI/BI Dashboards, Genie uses Agent mode by default. Workspace admins enable the feature in Workspace Previews, after which users turn on the Agent toggle in Genie spaces; API support and unstructured document analysis are planned additions.
Hanlin SunBanks don't have an AI problem — they have a data platform problem
The post argues that banks’ difficulty scaling AI stems less from model capability than from fragmented data, weak governance, and limited real-time access. Examples from CBA Live 2026 include model drift in credit scoring, fragmented customer signals, and a collections model that predicted, with 85% accuracy, how many days newly delinquent accounts would take to cure using governed data. It also presents Erica’s 3.2 billion interactions since 2018 as evidence that production AI requires continuous tuning, monitoring, and edge-case management, while generic models decay in frontline settings. The proposed Databricks architecture combines Lakehouse, Unity Catalog, Lakeflow, Lakebase, MLflow, Model Monitoring, Online Feature Store, Genie, and Agent Bricks to support governed analytics, low-latency decisions, auditable models, and constrained agent actions. Its conclusion is that a shared data platform should precede additional AI use cases because it improves deployment speed, trust, explainability, and regulatory defensibility.
Naeem Rehman, Jennifer MillerWhy your agents can't read enterprise documents — and how to fix it
Enterprise agents often reason well over clean text but struggle with scanned PDFs, inconsistent layouts, nested tables, images, and handwriting, making document processing an accuracy ceiling for business workflows. The post announces Document Intelligence, a pipeline of chainable AI Functions: ai_parse_document converts scans into layout-enriched structured text, while ai_classify and ai_extract support routing and entity extraction. On OfficeQA treasury-bond documents, preprocessing with ai_parse_document produced a 16% average performance gain across tested agent frameworks without changing their reasoning harness. Across invoice, contract, medical-note, and financial-filing extraction tasks, it reportedly delivered the highest accuracy at 5–7x lower cost than comparable pipelines, with serverless batch infrastructure scaling the same SQL call from 100 to 100,000. Databricks integrates ingestion, orchestration, governance, and agent building through Lakeflow Connect, Lakeflow Jobs or Spark Declarative Pipelines, Unity Catalog, and Agent Bricks, aiming to replace fragmented services with reusable workflows.
Archika Dogra, Sergei Tsarev, Erich ElsenBuilding with Databricks Document Intelligence and Lakeflow
The post presents a production-oriented approach to Intelligent Document Processing (IDP) that brings PDFs, images, and office documents into Databricks’ governed data lifecycle. Lakeflow Connect ingests files from sources such as SharePoint and Google Drive into Unity Catalog Volumes and tables, applying access control, lineage, and auditing while supporting incremental reads and writes. Databricks’ Document Intelligence then uses ai_parse_document to preserve structure in VARIANT output, with ai_extract, ai_classify, and ai_prep_search available for enrichment, routing, and retrieval preparation. Lakeflow Jobs productionizes the workflow with orchestration, control flow, retries, triggers, serverless scaling, and observability across ingestion, parsing, and serving tasks. The post also recommends joining extracted data to master data, using service principals, and monitoring drift and extraction accuracy, presenting the combined platform as a way to create trusted, queryable datasets from unstructured content.
Giselle Goicochea, Joanna Zouhour
