Loading…
Data Governance
42 posts about Data Governance. Every summary links to the original.
Machine Learning Use Cases: Practical Industry Applications
This guide surveys practical machine learning applications across industries, from medical imaging and fraud detection to demand forecasting, document processing, and customer-service automation. It defines supervised, unsupervised, semi-supervised, and reinforcement learning, then relates technique selection to the business question, data type, and availability of labels. Technical coverage includes convolutional neural networks for image analysis, transformers and large language models for generative AI, and time-series workflows using cleaning, lag features, and forecasting. The operational guidance addresses feature stores, experiment tracking, CI/CD, drift monitoring, retraining, cost optimization, fairness, privacy, explainability, and model risk management. Templates and checklists structure projects around business problems, data sources, metrics, architecture, measured outcomes, validation, governance, and escalation, while cited Databricks case studies and tools provide implementation examples.
Databricks StaffThe AI scaling gap hiding in digital native companies
An Economist survey of 1,220+ executives across eight industries finds that digital native companies lead in ambition and breadth of AI deployment, but not in full operational maturity. At 18%, they are the most likely group to prioritize embedding AI across core processes at scale, and nearly 92% report AI ROI ahead of plan. Yet they lead fully embedded AI—defined as use by 100+ users, SLAs, and performance and impact monitoring—in only one of eight functions, R&D/product development; they rank seventh in finance and sixth in operations and supply chain. Telecom, media and entertainment, manufacturing, and energy outperform them in selected functions despite lower stated scaling priority. The proposed response is to build shared, production-grade foundations for data, governance, workloads, models, agents, and applications, with security, lineage, monitoring, and performance measurement treated as reusable capabilities.
Madelyn MullenAI success starts with clean data, not just better models
Kraken’s data transformation work argues that successful AI depends on clean, unified, accessible data rather than model quality alone. Serving more than 90 million customer accounts across 27 countries, the platform uses Databricks to distribute data securely and at scale, while clients need documentation, join logic and business context to make it useful. Unification reduces the analyst bottleneck, builds trust in shared numbers and enables self-service analytics, including conversational querying through Databricks Genie. The discussion also describes metadata as a live model input: Unity Catalog and Delta Sharing let Kraken share context alongside data instead of relegating it to PDFs or separate web pages. Reported client examples include call-center dashboards updated every few hours with predictive models and faster tariff experimentation, while organizations with stronger data skills and culture are positioned to adopt agentic AI more quickly.
Aly McGueThe federal data paradox: Rich in data, poor in access
Federal agencies have invested substantially in data infrastructure, but program directors, policy analysts, oversight officials, and budget examiners often still rely on technical intermediaries to answer operational questions. The post frames this as the unresolved “last mile” of federal data modernization: data lakes, APIs, dashboards, evidence-based policymaking mandates, and agency CDO functions have advanced infrastructure without making it usable by most decision-makers. It presents Databricks Genie as a natural-language interface that lets staff query agency data in plain language, including questions requiring joins across disbursement, eligibility, and geographic data, while retaining existing access controls and policies. Genie runs on Unity Catalog with role-based access controls, audit logging, and data lineage; it also supports federated cross-agency queries and records queries, answers, and sources, which the post associates with oversight, FOIA readiness, and accountability.
Kacey HertanModel risk governance is not the same as risk intelligence
Financial institutions have invested heavily in model governance frameworks, stress testing infrastructure, limit monitoring, data feeds, and dashboards, but risk leaders may still lack fast access to what those models are telling them. When a CRO must assess credit concentration, scenario sensitivity, or relationships between market positions and credit exposures, answering can require navigation across model outputs, analyst interpretation, and disconnected data systems. The proposed approach uses Databricks AI/BI Genie to let leaders query risk data in natural language, while retaining access controls, audit logging, Unity Catalog lineage, cross-risk data, and stress-test outputs in one environment. The stated distinction is that governance establishes necessary controls, whereas conversational risk intelligence supports questions that fixed dashboards did not anticipate, including comparisons with internal limits.
Kim HattonTop Data Warehouse Tools For Modern Data Analytics
Data warehouse tools are presented as a strategic choice for analytics and ML teams facing fragmented estates of warehouses, lakes, and standalone ML systems. The guide proposes evaluating platforms across query performance, scalability, data integration, BI connectivity, total cost of ownership, and governance and security, with attention to MPP, columnar storage, and decoupled compute and storage. It contrasts traditional warehouses, data lakes, and lakehouses: warehouses favor structured SQL analytics, lakes offer native-format flexibility but weaker quality and performance guarantees, and lakehouses combine these capabilities using open formats such as Delta Lake and Apache Iceberg. The stated conclusion is that a lakehouse can provide a single governed foundation for SQL, BI, streaming, ML, and AI workloads, while teams should select tools according to workload, scale, budget, and future AI needs.
Databricks StaffUnlocking SAP business context in Databricks with semantic metadata Delta Sharing
SAP Business Data Cloud now offers generally available semantic metadata synchronization with Databricks Unity Catalog for mounted SAP BDC Delta Shares. When a table is accessed, Unity Catalog receives table-level business-friendly display names, descriptions, context, and governance tags, while SAP BDC remains the single source of truth and subsequent changes are reflected. The capability builds on BDC Connect and Delta Sharing, allowing governed SAP data products to be discovered and combined with other enterprise sources without recreating business context or governance separately. Column descriptions, table relationships such as primary and foreign keys, and PersonalData namespace tags give Databricks AI Assistant and AI/BI Genie explicit context for natural-language questions and join-ready queries. The stated result is more understandable, discoverable, and AI-ready SAP data, with automated classification signals supporting compliance, access control, and responsible AI.
Akram Chetibi, Katie Cummiskey, Moe Derakhshani, Abhijit ChakankarPredicting readmissions isn't enough. Acting in time is.
Readmission risk models can identify patients likely to return within 30 days, but prediction alone does not ensure timely intervention. In large health systems, risk information may remain in population-health dashboards or EHR flags without reaching the care coordinator with enough context to create an effective post-discharge plan. The post presents Databricks Genie as a natural-language interface for governed clinical and outcomes data, allowing leaders to query readmission rates and compare periods while combining EHR, operational, financial, risk-score, intervention, and outcome information. Genie operates within Unity Catalog governance, with access controls, audit logging, and de-identification policies, and is described as supporting EHR integration and clinical taxonomy awareness; the intended result is to shorten the path from prediction to action.
Adam CrownClinical trials run longer than they have to. That's a patient problem.
Clinical trial operations can lose weeks because site-level performance data is reviewed with a 2–4 week lag: enrollment velocity is monthly, while protocol deviation rates are assessed quarterly. The post presents Databricks Genie as a natural-language interface to unified trial data, allowing clinical operations leaders to query enrollment, screen-failure, protocol-deviation, query-response, and data-entry metrics across sites. Its stated capabilities include automatic site comparison, integration across CTMS, EDC, safety databases, and site-performance data, plus protocol-aware reasoning and traceability to source records for GCP documentation. The example query identifies Phase II oncology sites with screen-failure rates above 40% over 60 days and compares enrollment pace with activation targets, positioning earlier detection as a way to reduce timeline impact and speed treatment access.
Adam CrownInteroperability between Unity Catalog and Google BigQuery via catalog federation
Databricks and Google Cloud announce bidirectional catalog federation that lets customers access the same data from Unity Catalog or BigQuery without duplication. The interoperability uses Apache Iceberg REST catalog APIs, enabling both platforms to read and write the same tables through a common open interface. Google Cloud's catalog federation is in preview for reading Unity Catalog-managed tables from BigQuery, while Google Cloud's Lakehouse federation is in private preview for Databricks users to govern and read foreign Iceberg tables. Unity Catalog's policies, fine-grained access controls, and lineage tracking apply to federated data, and users can query it with Genie using natural language. The announcement positions open standards as a path to interoperability across data platforms and previews future catalog metadata exchange through Iceberg REST catalog APIs.
John Spencer, Jason ReidHow conversational analytics removes the BI bottleneck
The interview presents conversational analytics as a way to move beyond fixed dashboards and turn governed data into decisions and workflows. Databricks’ Genie lets nontechnical users ask open-ended questions in plain language across Lakebase, Lakehouse, and federated sources, while Unity Catalog applies shared definitions and access controls. Lakebase provides transactional storage, instant provisioning, environment forking without copying underlying data, and scale described as billions or trillions of records; examples include real-time matchmaking, routing, and purchase decisions. The discussion stresses that human oversight, refreshed data, business context, and controls against hallucination remain necessary, citing Fox Sports’ chatbot deployment as an example. It concludes that organizations combining trusted semantics with operational data systems can make faster decisions and automate work, while the competitive gap may compound for slower adopters.
Catherine BrownMultimodal Data Integration: Production Architectures for Healthcare AI
Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.
Maks KhomutskyiData Governance Platforms: Evaluation & Feature Guide
This guide presents a framework for evaluating data governance platforms for enterprise deployment, distinguishing governance—the policies, roles, and controls for data use—from data management, the operational execution of those policies. It recommends assessing metadata-centered capabilities including continuous data profiling, catalog search and enrichment, end-to-end lineage, RBAC and ABAC, sensitive-data detection, masking, audit trails, compliance reporting, and data-subject request workflows. Vendor assessment should also cover pre-built connectors, REST APIs and SDKs, near-real-time metadata synchronization, schema-drift handling, cross-cloud federation, usability, support, implementation timelines, and three-to-five-year total cost of ownership. The proposed decision process selects three leading candidates, pilots them with representative structured and unstructured datasets, defines quality, lineage, adoption, and exit metrics, and uses executive review before procurement; governance is framed as an ongoing program that expands with AI workloads and regulatory change.
Databricks StaffIntroducing the Databricks Excel add-in for business users
Databricks has introduced the Excel Add-in, now in public preview, to let business users import and analyze lakehouse data directly in Excel without writing SQL or configuring ODBC drivers. Built on Databricks SQL and Unity Catalog, it provides point-and-click access to tables and metric views, native Excel pivot tables, refreshable results, reusable Workspace queries, and optional saved SQL. Metric views let data teams define business semantics once while Unity Catalog centrally manages permissions and controls, helping avoid duplicated logic and metric drift. The preview supports selecting tables and metric views, parameterized custom functions, manual refreshes, pivot-table filtering, query reuse, and workspace changes; scheduled refresh, AI integrations, and other usability improvements are planned next.
Toussaint Webb, Kaitlin BaumgardnerBuilding an Enterprise Data Management Strategy
Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.
Databricks StaffAI Data Transformation Guide for Data Engineers and Data Scientists
AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.
Databricks StaffFrom Static Policies to Autonomous Insurance: How AI enables Real-Time Coverage
Insurance products often rely on annual pricing, periodic risk assessment, and coverage changes tied to major life events, even though driving behavior, digital fraud exposure, and travel risk can change in seconds. The post presents embedded insurance and usage-based insurance as a shift toward contextual, just-in-time coverage delivered through apps, checkout flows, vehicle purchases, and partner platforms. Its automotive example combines telematics, geospatial data, and real-time weather signals to update risk scores and premiums from trip behavior such as braking, following distance, mileage, location, and environmental conditions. A Databricks Lakehouse architecture registers devices, ingests telemetry through Zerobus, curates data in a governed Medallion architecture, and exposes features through Lakebase and Databricks One, with Unity Catalog supporting access controls, lineage, explainability, and audit trails. The proposed trajectory leads from embedded coverage toward autonomous insurance whose policies continuously adapt to risk.
Amee Vora, Anindita Mahapatra, Marcela GranadosBridging data science and marketing: Databricks unveils Delta Sharing integration for Adobe Experience Platform and agentic marketing workflows
Databricks and Adobe announce three partnership milestones aimed at connecting governed Lakehouse data with marketing activation in Adobe Experience Platform. The changes include Delta Sharing support for zero-copy access, a Databricks Genie connection to Adobe Marketing Agent through Model Context Protocol (MCP), and an upcoming Adobe Marketing Agent beta for the Databricks Marketplace. Using Unity Catalog as the source, Experience Platform can access shared tables and views as virtual tables, while Adobe Data Distiller queries live Databricks data without copying underlying records. Through MCP, agents in both platforms can use governed datasets, metadata, models, campaign metrics, SQL Warehouses, and Model Serving endpoints with natural-language interactions. The stated outcome is a shared, real-time context for audience creation, campaign analysis, closed-loop attribution, and model improvement, while reducing data movement, duplication, latency, egress, and governance overhead.
Justin Fenton, Dan Zuckerberg, Katy YuanBanks don't have an AI problem — they have a data platform problem
The post argues that banks’ difficulty scaling AI stems less from model capability than from fragmented data, weak governance, and limited real-time access. Examples from CBA Live 2026 include model drift in credit scoring, fragmented customer signals, and a collections model that predicted, with 85% accuracy, how many days newly delinquent accounts would take to cure using governed data. It also presents Erica’s 3.2 billion interactions since 2018 as evidence that production AI requires continuous tuning, monitoring, and edge-case management, while generic models decay in frontline settings. The proposed Databricks architecture combines Lakehouse, Unity Catalog, Lakeflow, Lakebase, MLflow, Model Monitoring, Online Feature Store, Genie, and Agent Bricks to support governed analytics, low-latency decisions, auditable models, and constrained agent actions. Its conclusion is that a shared data platform should precede additional AI use cases because it improves deployment speed, trust, explainability, and regulatory defensibility.
Naeem Rehman, Jennifer MillerIntroducing the Databricks Connector for Google Sheets: Real-Time, Governed Lakehouse Data in the Sheets Users Love
Databricks has introduced a Google Sheets connector that brings live, governed lakehouse data into a familiar spreadsheet environment. The connector addresses reliance on CSV exports, snapshots, and custom pipelines, which left business teams with siloed copies, inconsistent metrics, and slower decisions. Built on Databricks SQL and Unity Catalog, it lets users query governed datasets, including Unity Catalog Metric Views, through a no-code GUI or SQL, while Unity Catalog manages permissions. Users can refresh results manually or schedule refreshes, keeping Sheets data current without exporting snapshots. The connector is Generally Available to all Databricks customers, and Nubank reports that it has enabled non-technical users to explore data in Sheets while preserving control.
Toussaint Webb
