Loading…
Topic
135 posts about Unity Catalog. Every summary links to the original.
See the idea
An agentic ad stack wires governed customer signals to campaign execution through a shared tool interface and server-side guardrails. The diagram shows each layer, the direction of a tool call, and where control stays human.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.
Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.
Automatic Upgrades gives Unity Catalog managed tables a hands-off path to adopting best-practice table features without manually auditing thousands of tables or running ALTER TABLE repeatedly. It observes table access over a 100-day window, verifies that every accessing Databricks client supports the feature, skips inactive or externally accessed tables, and applies eligible changes through a lightweight background job. New tables inherit a feature when it becomes a schema default, while explicit table properties take precedence. Supported capabilities include Row Tracking, Automatic Liquid Clustering, Deletion Vectors, Column Mapping, Parquet V2, Catalog Commits, and Checkpoint V2, with benefits spanning query speed, storage and write costs, interoperability, and reliability. Changes are visible in DESCRIBE HISTORY and Catalog Explorer, reversible per table, and not re-enabled after a user disables one.
Vibe Data Modeling is presented as a way to build a governed Silver-layer data model from a plain-English description of a business, addressing the months or years often required to hand-build one or adapt a generic industry template. The single-notebook implementation uses four widgets and a multi-model LLM agent to organize organizations into divisions, domains, subdomains, products, and attributes, then generate a logical model and deploy it to Unity Catalog. Its pipeline advances through four generate-and-validate stages, applying 251 rules, deterministic structural gates, two architect reviews, and a retry loop that changes strategy when checks fail. The authoritative model.json also produces schemas, Delta tables, informational foreign keys, classification tags, metric views, an RDFS ontology, DBML, and synthetic sample data, while plain-English refinements create auditable, reversible versions.
At Data + AI Summit 2026, Databricks and OpenAI presented a partnership centered on combining OpenAI’s frontier models and agents with Databricks’ enterprise context and control. The post describes GPT models and Codex running natively on Databricks, with Unity AI Gateway governing interactions, budgets, routing, auditing, and visibility, while Agent Tools provide governed enterprise data access through MCPs. Examples include OpenAI’s marketing data foundation on Databricks, which cut storage costs by $400,000 per month, and a Hertz application built in 11 business days with GPT-5.5 and Databricks that lifted lead conversion from 60–65% to 75–80%. The sessions argue that deployment, security, evaluation, monitoring, context, and sharing comprise most of the work around enterprise agents. A joint virtual event scheduled for August will address shipping agentic applications at scale, including a Stellantis case study.
Databricks Forward Deployed Engineering introduces Decision Execution Platforms (DEPs), an enterprise analytics category intended to connect KPI signals, executive decisions, operational execution, and measured outcomes. The proposal addresses workflows in which dashboards reveal problems but meetings, decks, spreadsheets, and messaging threads leave implementation fragmented and impact measurement disconnected. DEPs run the four-stage loop on governed Databricks infrastructure: agents recommend actions, alternatives, predicted impact, and reasoning; approved choices execute through systems of record; and results persist in a Decision Log for continuous learning. Their architecture combines a foundation of Lakebase, Genie, Unity Catalog, Lakehouse, Agent Bricks, and MLflow with an SDK of reusable primitives and a Databricks Apps executive surface. A retailer case used a DEP to unify fulfillment data and enable simulated, controlled rerouting, with scaling aimed at measurable bottom-line and customer-satisfaction outcomes.
An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.
The Office for Students, which regulates more than 400 higher education providers in England, modernised its data and analytics environment after a legacy platform could no longer handle growing volumes, varied sources, or emerging analytical demands. Its data spans millions of student records collected over 15 to 20 years, and a workflow processing about 300 million records took eight hours. Moving to Databricks consolidated structured, qualitative, and near-live data with analytics and AI workflows, while Unity Catalog added lineage, access controls, and security patterns for governed use. Genie Code reduced a student segmentation analysis from at least two weeks for two analysts to half a day, and a provider-registration triage proof of concept flags missing submissions earlier. The organisation frames AI as decision support rather than decision-making, keeping humans responsible for regulatory judgments.
AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.
Serverless databases are presented as a baseline for AI applications whose traffic can be volatile, idle for long periods, or spike when agents fan out queries. The guide distinguishes managed serverless systems from autoscaling products by focusing on architectural separation of compute and storage, true scale-to-zero, cold-start behavior, connection handling, pricing, performance, portability, governance, and AI capabilities such as vector search. It recommends evaluating both low- and high-utilization costs, published warm-up times, tail latency (p95/p99), and built-in pooling or HTTP/Data APIs for high-concurrency agents and serverless functions. The text positions Lakebase as an example that combines serverless Postgres, shared lakehouse storage, and Unity Catalog governance, and cites reported cost and management reductions from a 2025 study while noting that provisioned deployments may suit continuously high-throughput workloads.
Professional basketball’s Hawk-Eye SkeleTRACK feed produces roughly 22,620 positional updates per second—about 65 million records per 48-minute game—yet teams often cannot turn that volume into timely, trusted decisions. The integration gap comes from separate vendors for tracking, wearables, video, scouting, and medical data, alongside calibration differences, weak provenance, and compute limits. The Databricks Data + AI Platform is presented as a governed lakehouse that ingests feeds with Lakeflow, refines them through medallion layers, and uses Unity Catalog for lineage, access control, and auditing. Models for shot probability, injury risk, and fatigue can run alongside serving and custom applications, with Lakebase supporting sub-second interactive queries. Applications include proactive load management, real-time coaching intelligence, and enriched broadcast or fan experiences across tracking-rich sports.
Daikin Applied Americas needed to scale reliable data pipelines across growing analytics and AI use cases involving operational, manufacturing, and service data while coordinating development across teams. It adopted Databricks Genie Code within a structured operating model, using Unity Catalog context, reusable MECE skills, and explicit checkpoints across Bronze, Silver, and Gold layers to guide planning and execution. The framework defines competencies such as source grain, transformation patterns, canonical alignment, governance, and business-entity modeling, moving standards out of long prompts and into the development environment. The team reports that pipelines that once took days to prototype could be generated in minutes, with faster iteration, more consistent outputs, less structural correction, reduced architectural drift, and greater trust in AI-assisted results.
Kythera Labs is building an AI-native healthcare strategy platform on Databricks to give health systems access to strategic intelligence that historically required specialized analysts or consulting firms. Its foundation converts 339 billion medical and prescription claims covering more than 300 million patients into governed, event-based data, resolving providers, harmonizing codes across 130 vocabularies, and reconstructing patient journeys. Healthcare Strategy Agent, built with Agent Bricks, lets executives ask questions such as where oncology referrals are going and receive analyses of leakage, competing providers, physicians, and reimbursement opportunity in minutes. A Louisiana health system went live within ten days and reported 150% greater visibility into encounters, 12% more keepage, 22% less leakage, and $3.8 million in estimated annualized retained-encounter value. Unity Catalog, Lakebase, Delta Lake, Delta Sharing, and serverless infrastructure provide shared governance, lineage, access controls, and operational integration.
Databricks says Gartner positioned it highest for execution and furthest for vision in the Magic Quadrant for the second consecutive year. The post connects this recognition to a category reclassified from “Data Science and Machine Learning” to “AI Platforms for Data Science and Machine Learning,” and argues that agentic applications require enterprise data, governance, observability, and business context. Databricks presents a unified approach combining the lakehouse, Lakebase, Agent Bricks, Unity Catalog, and Unity AI Gateway to build, monitor, and govern agents, models, data, apps, and tools. Reported examples include YipitData’s 20x increase in company coverage with 92–95% tagging accuracy, Block’s unified AI and data estate, and Novo Nordisk’s attribution of more than $157 million in net new value to governed clinical-trial optimization.
Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.
Data scientists connect statistics, programming, and domain knowledge to turn raw data into models, recommendations, and decisions with business consequences. The role now extends beyond classical modeling to large language models, generative AI applications, agentic systems, and production workflows involving deployment, monitoring, and retraining. Modern practice combines Python, SQL, and libraries such as scikit-learn, PyTorch, Spark, and MLflow with data engineering basics, communication, and domain expertise. Data scientists contribute across problem framing, data access, preparation, feature engineering, experimentation, deployment, and lifecycle monitoring, with governed access and lineage helping align training data with production data. The article concludes that AI assistants and agents can automate routine work, but human judgment remains necessary for trustworthy evaluation, business framing, and actionable recommendations.
Stagwell built a Databricks Marketplace App to let brands match first-party records against its Identity Spine without sending raw customer data outside their own workspaces. It combines Databricks Clean Rooms, Unity Catalog, Jobs and Notebooks, Delta Sharing, and a React/Express interface. Workspace OAuth and On-Behalf-Of authorization enforce the user's Unity Catalog permissions, while separate service principals and a scoped personal access token support app and clean-room operations. Within the clean room, a packaged notebook joins brand data with the Identity Spine, resolves multiple identifiers, and computes match rates, coverage metrics, and household and consumer IDs; results include aggregated audience insights for activation, while raw customer data and proprietary matching code remain separated.
Row-level security (RLS) is a database access control that filters which table rows a user may read or change according to identity, role, or session context. It supports multi-tenant SaaS, regional segregation, departmental access, compliance, healthcare, and partner use cases, while typically layering table-level grants with column-level security or data masking. The database applies a policy or predicate automatically at query time across BI dashboards, notebooks, ad-hoc SQL, APIs, and third-party tools, with separate USING and WITH CHECK rules for reads and writes. On Databricks, Unity Catalog row filters use SQL user-defined functions and apply across Databricks SQL, notebooks, jobs, and connected BI tools. The source recommends simple predicates, indexed keying columns, identity-based testing, and explicit enforcement for both reads and writes.
Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.
Traditional customer data platforms (CDPs) were built for human-managed, batch-based campaigns, but the post argues that agentic buying requires millisecond speed, hyper-personalization, and richer context. It contrasts the familiar Golden Record with Golden Context, which combines customer data with current business goals and the history and outcomes of prior decisions. The proposed Agentic CDP uses “Infinity Campaigns,” always-on engagement loops that use LLMs and agents to adapt messaging, timing, and channels for individuals. It is also embedded in the data foundation, bringing customer, business, and decision context together under existing governance, and is designed for agents and humans from the outset. Databricks presents CustomerLake as an implementation of these principles for its platform.
The best engineering reads, once a week.
