Loading…
Topic
79 posts about Data Governance. Every summary links to the original.
See the idea
Business Intelligence turns operational data into decisions, but every speed-up trades against trust. This diagram maps the benefit, cost, and condition behind governed metrics, shared context, and provenance.
Asset management runs as a layered control plane: custodian and pricing data feed a governed context layer, agents run the overnight fund work, and a human signs off before anything is published.
Boards of directors turn ownership into a governance control loop: shareholders elect directors, directors set strategy and guardrails, committees verify independently, and management executes inside those limits.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
SAP Business Data Cloud now offers generally available semantic metadata synchronization with Databricks Unity Catalog for mounted SAP BDC Delta Shares. When a table is accessed, Unity Catalog receives table-level business-friendly display names, descriptions, context, and governance tags, while SAP BDC remains the single source of truth and subsequent changes are reflected. The capability builds on BDC Connect and Delta Sharing, allowing governed SAP data products to be discovered and combined with other enterprise sources without recreating business context or governance separately. Column descriptions, table relationships such as primary and foreign keys, and PersonalData namespace tags give Databricks AI Assistant and AI/BI Genie explicit context for natural-language questions and join-ready queries. The stated result is more understandable, discoverable, and AI-ready SAP data, with automated classification signals supporting compliance, access control, and responsible AI.
Readmission risk models can identify patients likely to return within 30 days, but prediction alone does not ensure timely intervention. In large health systems, risk information may remain in population-health dashboards or EHR flags without reaching the care coordinator with enough context to create an effective post-discharge plan. The post presents Databricks Genie as a natural-language interface for governed clinical and outcomes data, allowing leaders to query readmission rates and compare periods while combining EHR, operational, financial, risk-score, intervention, and outcome information. Genie operates within Unity Catalog governance, with access controls, audit logging, and de-identification policies, and is described as supporting EHR integration and clinical taxonomy awareness; the intended result is to shorten the path from prediction to action.
Clinical trial operations can lose weeks because site-level performance data is reviewed with a 2–4 week lag: enrollment velocity is monthly, while protocol deviation rates are assessed quarterly. The post presents Databricks Genie as a natural-language interface to unified trial data, allowing clinical operations leaders to query enrollment, screen-failure, protocol-deviation, query-response, and data-entry metrics across sites. Its stated capabilities include automatic site comparison, integration across CTMS, EDC, safety databases, and site-performance data, plus protocol-aware reasoning and traceability to source records for GCP documentation. The example query identifies Phase II oncology sites with screen-failure rates above 40% over 60 days and compares enrollment pace with activation targets, positioning earlier detection as a way to reduce timeline impact and speed treatment access.
Grab’s Signals Marketplace uses three foundational platforms to make data certification operational across its data mesh: Hubble, Genchi, and the Data Contract Registry. Hubble, built on DataHub, catalogs assets, ownership, documentation, lineage, contracts, and certification status, while its event-driven engine recalculates certification as metadata and upstream health change. Genchi provides self-service observability and tests for freshness, volume, schema, completeness, and business checks; the registry manages explicit expectations between producers and consumers. Together, these signals let Hubble classify assets as Uncertified, Certified, CertifiedPlus, or Revoked, giving consumers a workflow centered on certified data rather than hearsay. The post reports that P80 datasets, the tables accounting for 80% of queries, fell by over 58% in one year after the Signals Marketplace campaign began in 2024, and positions the foundation as support for AI agents and automated analytics.
Databricks and Google Cloud announce bidirectional catalog federation that lets customers access the same data from Unity Catalog or BigQuery without duplication. The interoperability uses Apache Iceberg REST catalog APIs, enabling both platforms to read and write the same tables through a common open interface. Google Cloud's catalog federation is in preview for reading Unity Catalog-managed tables from BigQuery, while Google Cloud's Lakehouse federation is in private preview for Databricks users to govern and read foreign Iceberg tables. Unity Catalog's policies, fine-grained access controls, and lineage tracking apply to federated data, and users can query it with Genie using natural language. The announcement positions open standards as a path to interoperability across data platforms and previews future catalog metadata exchange through Iceberg REST catalog APIs.
The interview presents conversational analytics as a way to move beyond fixed dashboards and turn governed data into decisions and workflows. Databricks’ Genie lets nontechnical users ask open-ended questions in plain language across Lakebase, Lakehouse, and federated sources, while Unity Catalog applies shared definitions and access controls. Lakebase provides transactional storage, instant provisioning, environment forking without copying underlying data, and scale described as billions or trillions of records; examples include real-time matchmaking, routing, and purchase decisions. The discussion stresses that human oversight, refreshed data, business context, and controls against hallucination remain necessary, citing Fox Sports’ chatbot deployment as an example. It concludes that organizations combining trusted semantics with operational data systems can make faster decisions and automate work, while the competitive gap may compound for slower adopters.
Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.
This guide presents a framework for evaluating data governance platforms for enterprise deployment, distinguishing governance—the policies, roles, and controls for data use—from data management, the operational execution of those policies. It recommends assessing metadata-centered capabilities including continuous data profiling, catalog search and enrichment, end-to-end lineage, RBAC and ABAC, sensitive-data detection, masking, audit trails, compliance reporting, and data-subject request workflows. Vendor assessment should also cover pre-built connectors, REST APIs and SDKs, near-real-time metadata synchronization, schema-drift handling, cross-cloud federation, usability, support, implementation timelines, and three-to-five-year total cost of ownership. The proposed decision process selects three leading candidates, pilots them with representative structured and unstructured datasets, defines quality, lineage, adoption, and exit metrics, and uses executive review before procurement; governance is framed as an ongoing program that expands with AI workloads and regulatory change.
Databricks has introduced the Excel Add-in, now in public preview, to let business users import and analyze lakehouse data directly in Excel without writing SQL or configuring ODBC drivers. Built on Databricks SQL and Unity Catalog, it provides point-and-click access to tables and metric views, native Excel pivot tables, refreshable results, reusable Workspace queries, and optional saved SQL. Metric views let data teams define business semantics once while Unity Catalog centrally manages permissions and controls, helping avoid duplicated logic and metric drift. The preview supports selecting tables and metric views, parameterized custom functions, manual refreshes, pivot-table filtering, query reuse, and workspace changes; scheduled refresh, AI integrations, and other usability improvements are planned next.
Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.
AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.
Insurance products often rely on annual pricing, periodic risk assessment, and coverage changes tied to major life events, even though driving behavior, digital fraud exposure, and travel risk can change in seconds. The post presents embedded insurance and usage-based insurance as a shift toward contextual, just-in-time coverage delivered through apps, checkout flows, vehicle purchases, and partner platforms. Its automotive example combines telematics, geospatial data, and real-time weather signals to update risk scores and premiums from trip behavior such as braking, following distance, mileage, location, and environmental conditions. A Databricks Lakehouse architecture registers devices, ingests telemetry through Zerobus, curates data in a governed Medallion architecture, and exposes features through Lakebase and Databricks One, with Unity Catalog supporting access controls, lineage, explainability, and audit trails. The proposed trajectory leads from embedded coverage toward autonomous insurance whose policies continuously adapt to risk.
Databricks and Adobe announce three partnership milestones aimed at connecting governed Lakehouse data with marketing activation in Adobe Experience Platform. The changes include Delta Sharing support for zero-copy access, a Databricks Genie connection to Adobe Marketing Agent through Model Context Protocol (MCP), and an upcoming Adobe Marketing Agent beta for the Databricks Marketplace. Using Unity Catalog as the source, Experience Platform can access shared tables and views as virtual tables, while Adobe Data Distiller queries live Databricks data without copying underlying records. Through MCP, agents in both platforms can use governed datasets, metadata, models, campaign metrics, SQL Warehouses, and Model Serving endpoints with natural-language interactions. The stated outcome is a shared, real-time context for audience creation, campaign analysis, closed-loop attribution, and model improvement, while reducing data movement, duplication, latency, egress, and governance overhead.
The post argues that banks’ difficulty scaling AI stems less from model capability than from fragmented data, weak governance, and limited real-time access. Examples from CBA Live 2026 include model drift in credit scoring, fragmented customer signals, and a collections model that predicted, with 85% accuracy, how many days newly delinquent accounts would take to cure using governed data. It also presents Erica’s 3.2 billion interactions since 2018 as evidence that production AI requires continuous tuning, monitoring, and edge-case management, while generic models decay in frontline settings. The proposed Databricks architecture combines Lakehouse, Unity Catalog, Lakeflow, Lakebase, MLflow, Model Monitoring, Online Feature Store, Genie, and Agent Bricks to support governed analytics, low-latency decisions, auditable models, and constrained agent actions. Its conclusion is that a shared data platform should precede additional AI use cases because it improves deployment speed, trust, explainability, and regulatory defensibility.
Databricks has introduced a Google Sheets connector that brings live, governed lakehouse data into a familiar spreadsheet environment. The connector addresses reliance on CSV exports, snapshots, and custom pipelines, which left business teams with siloed copies, inconsistent metrics, and slower decisions. Built on Databricks SQL and Unity Catalog, it lets users query governed datasets, including Unity Catalog Metric Views, through a no-code GUI or SQL, while Unity Catalog manages permissions. Users can refresh results manually or schedule refreshes, keeping Sheets data current without exporting snapshots. The connector is Generally Available to all Databricks customers, and Nubank reports that it has enabled non-technical users to explore data in Sheets while preserving control.
The post argues that financial-services AI adoption is widespread, but execution—not model capability or strategy—is determining who captures value in 2026. It attributes stalled pilots to fragmented legacy infrastructure, inconsistent data, weak lineage, and insufficient control for governed, real-time workflows such as fraud detection, pricing, and personalization. Firms advancing further treat data as a managed asset, embed governance in data and model pipelines, and align data, analytics, and AI teams around shared definitions, workflows, and metrics. The proposed remedy is a unified lakehouse environment combining storage, compute, governance, lifecycle management, orchestration, streaming, and AI workflows, with Unity Catalog providing centralized access control, lineage, and auditing. The conclusion is that by the end of 2026, firms that embed AI into operational decisioning at scale will pull ahead of organizations still running pilots.
Grab began its Signals Marketplace data mesh journey in 2024 because its expanding, varied data ecosystem had ownership gaps, duplicated pipelines, central Data Engineering bottlenecks, broken downstream dependencies, inconsistent definitions, and uneven practitioner expertise. The decentralised model treats data as a product owned by domain teams, with Business Data Owners and Technical Data Owners accountable for upkeep, documentation, usability, and SLOs. Certification formalises expectations through data contracts covering schema, freshness, completeness, retention, change notice, and communication, while automated Data Production Incidents flag breached reliability guarantees for investigation. Using the percentage of queries hitting certified assets as its north star metric, Grab reports 75% of queries targeting certified assets, a 400% year-over-year increase in deprecated tables, and a 58% reduction in P80 datasets.
Metasense V2 addresses automated data governance by improving an LLM-powered metadata generation and classification system. It builds on an initial rollout that scanned more than 20,000 data entries at 300–400 entities per day, but retained human verification to catch misclassifications. Post-rollout owner feedback and manual classifications from the Data Governance Office supplied training and testing data, exposing failures involving PII in business email columns, nested JSON, and passenger communications. The revised approach splits PII and non-PII tagging, reduces the PII tag set from 21 to 8, shortens the prompt from 1,254 to 737 words, and divides tables exceeding 150 columns. LangChain and LangSmith support prompt development, evaluation across accuracy, latency, and error rate, deployment, and monitoring, while threshold-based alerts trigger a model-improvement protocol when misclassification rates rise.
Global applications can move customer data across borders, making data residency a compliance concern for organizations serving multinational users. The post defines data residency as legal requirements governing where data may be stored and processed, then distinguishes it from data sovereignty’s jurisdictional reach and data localization’s requirement that storage and processing remain within a jurisdiction. It notes that GDPR and CCPA focus on compliant handling and individual rights rather than necessarily mandating storage locations, while China’s Personal Information Protection Law requires certain Chinese citizens’ PII to be stored within China. The proposed approach uses cloud regions to align storage with local requirements, citing Heroku Enterprise Private Dynos across ten regions and Heroku Shield Private Spaces’ PCI, HIPAA, ISO, and SOC certifications; the post concludes that these features can make compliance more achievable while improving control and potentially application performance.
The best engineering reads, once a week.
