Loading…
Topic
135 posts about Unity Catalog. Every summary links to the original.
See the idea
An agentic ad stack wires governed customer signals to campaign execution through a shared tool interface and server-side guardrails. The diagram shows each layer, the direction of a tool call, and where control stays human.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Public-sector agencies face fraud methods including synthetic identities, deepfake-enhanced documentation, and personalized social engineering, while legacy controls remain fragmented and difficult to scale. The fictional Services Bureau demonstrates an operating model that combines Databricks Apps, Lakebase, Unity Catalog, Delta Sharing, Agent Bricks, and AI/BI Genie in a single fraud-operations environment. Governed data lands in Delta tables; Unity Catalog applies attribute-based access control, masks PII by role, and provides lineage, while agents connect live lakehouse queries, agency policies, and external fraud signals through MCP. Analysts review evidence and recommendations, then approve, override, or escalate cases, keeping human judgment central. The described workflow turns weeks of manual investigation into a day, supports dashboards and conversational SQL, and is presented as making fraud decisions faster, more secure, transparent, and defensible.
Genie Code is presented as a natural-language assistant for developing, orchestrating, deploying, and debugging data pipelines and jobs. It uses pipeline and job context, including code, configuration, run results, Unity Catalog metadata, lineage, popularity, and code samples, to help engineers discover datasets and understand data flows. Engineers can describe pipelines or jobs, and Genie Code can generate Spark Declarative Pipelines with Bronze, Silver, and Gold layers, sources, transformations, data quality expectations, and outputs, then configure orchestration, schedules, dependencies, Auto Loader, AutoCDC flows, and Declarative Automation Bundles. It also analyzes failures and unexpected row-count or schema changes, proposes cross-file updates with reviewable diffs, and supports extensions through custom instructions, agent skills, and MCP servers. The stated result is faster development and guided debugging while workflows remain aligned with Unity Catalog governance, performance, and data quality standards; future plans include background failure response and cluster right-sizing.
Snapchat Conversions API is now available as a Databricks Marketplace listing that lets teams activate first-party conversion signals directly from governed Lakehouse data. The included pre-built notebook reads gold-layer tables, batches web, app, and offline events, and supports deduplication with Snap Pixel, allowing server-side signals without double-counting. Because it runs in the Databricks environment, Unity Catalog preserves governance, lineage, security, and access controls while teams control what data is shared and when. The post says fuller server-side coverage can raise Event Match Quality, while citing that advertisers who improved their EQS from “Poor” to “Good” saw 26% higher ROAS and 49% lower CPI. It also describes fresher custom audiences and Lookalikes plus more complete deduplicated measurement.
The next generation of Databricks Genie expands the assistant beyond individual Genie Spaces, combining structured and unstructured enterprise data in a unified chat experience. It reuses logic from certified Genie Spaces, governed dashboards, and Databricks Apps, with metadata-based routing that prioritizes higher-trust sources, while new reasoning models and agent architecture handle questions spanning multiple domains. Built-in connectors for Google Drive and SharePoint, plus MCP support, let Genie access knowledge stores and take actions; Unity Catalog AI Gateway manages these connections. The experience replaces Databricks One as an account-level global home, adding domains, custom URLs, unified login, automated identity management, and governance through Unity Catalog. Native iOS and Android apps extend access to dashboards, apps, and chat beyond desktop environments.
The post presents the April 17, 2026 revision to model risk management guidance as a shift toward risk-based, principles-driven oversight, replacing several earlier issuances from the Federal Reserve, FDIC, and OCC. It identifies five operational changes: materiality tiering, end-to-end lifecycle governance, reproducible effective challenge, continuous drift and performance monitoring, and application of MRM principles to GenAI and agentic systems. Its proposed response is a Databricks reference architecture built on Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines, Feature Store, MLflow, Model Registry, Model Serving, and assurance tools. The design makes tiering metadata-driven and turns lineage, validation, approvals, monitoring, documentation, and retirement records into evidence generated during normal model work. The stated conclusion is that a unified substrate can reduce integration and manual evidence work, allowing regulatory changes to become configuration exercises rather than multi-quarter programs.
Databricks announces the Public Preview of Lakeflow Designer, a visual, no-code, AI-native experience for data preparation and analytics built directly into the Databricks platform. It targets analysts, domain experts, and other less technical users with drag-and-drop workflows and natural-language authoring, while representing each transformation as an operator with previews. Genie Code can use Unity Catalog metadata, descriptions, lineage, popularity, and example queries to generate or modify workflows, inspect transformation results, and iterate when needed. Every visual transformation produces production-ready Python code that can be reviewed, versioned in Git, and scheduled through Lakeflow Jobs, while users pay for compute rather than per-user licenses. The preview is available in all workspaces, although an administrator may need to enable Visual data prep in the preview portal.
Enterprise document workflows often require people to read PDFs, copy fields into spreadsheets, and re-enter data into business systems, creating delays, error risk, and weak auditability. The proposed alternative combines AI/BI Genie for natural-language queries over Delta tables, Agent Bricks for production agents, and Unity Catalog for permissions, lineage, and traceability. A five-phase workflow extracts fields into Lakeflow Bronze, Silver, and Gold layers, then supports structured queries, clause-level RAG answers, multi-agent routing, and actions through MCP servers. Examples include syncing validated rights data to ERP or CRM systems, updating billing entitlements, and triggering workflows from detected obligations. The approach targets media, advertising, ad tech, and telecommunications, where customers see improvements including faster month-end close, recovered revenue, reduced leakage, and lower operational risk.
Financial-services CFOs are being asked to act as Strategists and Catalysts, but fragmented legacy systems, T+1 batch processing, opaque reporting logic, and mismatched business semantics keep them focused on stewardship and retrospective operations. The proposed answer is Databricks as a unified, governed platform combining real-time streaming, centralized lineage, data and AI, with Unity Catalog, Lakeflow, Genie, and Agent Bricks addressing trust, latency, access, and model reproducibility. Unity Catalog can connect semantic definitions and end-to-end lineage from transactions through regulatory reports and models, while Lakeflow supports continuous ledger and liquidity processing. The post cites a global bank reducing liquidity-reporting processing from 10 hours to 8 minutes and Nationwide Insurance reporting a 5-point combined-ratio improvement and 3-point expense-ratio improvement. It presents a Lakehouse-based CFO stack as a shift from reporting historical results toward real-time capital management and previews AI-driven deposit and PPNR modeling.
Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.
Databricks has introduced the Excel Add-in, now in public preview, to let business users import and analyze lakehouse data directly in Excel without writing SQL or configuring ODBC drivers. Built on Databricks SQL and Unity Catalog, it provides point-and-click access to tables and metric views, native Excel pivot tables, refreshable results, reusable Workspace queries, and optional saved SQL. Metric views let data teams define business semantics once while Unity Catalog centrally manages permissions and controls, helping avoid duplicated logic and metric drift. The preview supports selecting tables and metric views, parameterized custom functions, manual refreshes, pivot-table filtering, query reuse, and workspace changes; scheduled refresh, AI integrations, and other usability improvements are planned next.
Mercedes-Benz built a cross-cloud data mesh to share after-sales data between AWS and Azure while supporting its data-defined vehicle strategy. Existing direct queries incurred egress costs, and weekly full loads delayed updates for datasets that included vehicle over-the-air events and workshop visits; source tables were in Iceberg while many consumers expected Delta. Using Unity Catalog and Delta Sharing, the company federated AWS Glue tables, established cross-metastore shares, and used periodic serverless Sync Jobs with Delta Deep Clone to incrementally replicate selected tables into Azure object storage for local querying. This hybrid model preserves direct sharing for fresher workloads while reducing transfer for less time-sensitive ones. For the initial ten data products, egress costs fell 66%, update frequency improved from weekly to every second day, and the source reports an approximately 93% annual reduction in its 50-use-case comparison.
Databricks introduces Agent mode in Genie spaces, an agentic process that plans, explores, and reasons over data to answer business questions. For each question, Genie can confirm an issue, test potential contributors through multiple queries, reflect on results, and decide what to investigate next, using Unity Catalog metadata and author-defined semantics from the Genie space. It produces a findings report with quantified results, identified contributors, visualizations, and references to the underlying SQL, and may include actionable recommendations. The agent scales its reasoning to task complexity, using validation for straightforward questions and more planning for multi-step investigations; in AI/BI Dashboards, Genie uses Agent mode by default. Workspace admins enable the feature in Workspace Previews, after which users turn on the Agent toggle in Genie spaces; API support and unstructured document analysis are planned additions.
The post argues that banks’ difficulty scaling AI stems less from model capability than from fragmented data, weak governance, and limited real-time access. Examples from CBA Live 2026 include model drift in credit scoring, fragmented customer signals, and a collections model that predicted, with 85% accuracy, how many days newly delinquent accounts would take to cure using governed data. It also presents Erica’s 3.2 billion interactions since 2018 as evidence that production AI requires continuous tuning, monitoring, and edge-case management, while generic models decay in frontline settings. The proposed Databricks architecture combines Lakehouse, Unity Catalog, Lakeflow, Lakebase, MLflow, Model Monitoring, Online Feature Store, Genie, and Agent Bricks to support governed analytics, low-latency decisions, auditable models, and constrained agent actions. Its conclusion is that a shared data platform should precede additional AI use cases because it improves deployment speed, trust, explainability, and regulatory defensibility.
Enterprise agents often reason well over clean text but struggle with scanned PDFs, inconsistent layouts, nested tables, images, and handwriting, making document processing an accuracy ceiling for business workflows. The post announces Document Intelligence, a pipeline of chainable AI Functions: ai_parse_document converts scans into layout-enriched structured text, while ai_classify and ai_extract support routing and entity extraction. On OfficeQA treasury-bond documents, preprocessing with ai_parse_document produced a 16% average performance gain across tested agent frameworks without changing their reasoning harness. Across invoice, contract, medical-note, and financial-filing extraction tasks, it reportedly delivered the highest accuracy at 5–7x lower cost than comparable pipelines, with serverless batch infrastructure scaling the same SQL call from 100 to 100,000. Databricks integrates ingestion, orchestration, governance, and agent building through Lakeflow Connect, Lakeflow Jobs or Spark Declarative Pipelines, Unity Catalog, and Agent Bricks, aiming to replace fragmented services with reusable workflows.
The post presents a production-oriented approach to Intelligent Document Processing (IDP) that brings PDFs, images, and office documents into Databricks’ governed data lifecycle. Lakeflow Connect ingests files from sources such as SharePoint and Google Drive into Unity Catalog Volumes and tables, applying access control, lineage, and auditing while supporting incremental reads and writes. Databricks’ Document Intelligence then uses ai_parse_document to preserve structure in VARIANT output, with ai_extract, ai_classify, and ai_prep_search available for enrichment, routing, and retrieval preparation. Lakeflow Jobs productionizes the workflow with orchestration, control flow, retries, triggers, serverless scaling, and observability across ingestion, parsing, and serving tasks. The post also recommends joining extracted data to master data, using service principals, and monitoring drift and extraction accuracy, presenting the combined platform as a way to create trusted, queryable datasets from unstructured content.
The best engineering reads, once a week.
