Loading…
Topic
80 posts about Data Governance. Every summary links to the original.
See the idea
Antimicrobial resistance surveillance is a layered data architecture: raw reads and demand signals flow in, get normalized into governed tables, and answer plain-language queries. The diagram maps each boundary.
BYOK lets a system run on assets you supply — your model credential or your private knowledge — instead of the vendor's defaults. This walkthrough traces six steps from registration to an audited response.
Business Intelligence turns operational data into decisions, but every speed-up trades against trust. This diagram maps the benefit, cost, and condition behind governed metrics, shared context, and provenance.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Financial institutions often split portfolio, operations, actuarial, and finance work across separate systems, even when teams depend on the same portfolios, positions, liabilities, and transactions. Databricks presents a unified, governed platform combining Databricks SQL, Unity Catalog Metric Views, Lakebase, Genie, Databricks Apps, and Lakeflow Designer to connect analysis with operational action. A worked workflow follows an actuary's duration analysis, a portfolio manager's scenario modeling and mandate changes, operations' IBOR/ABOR reconciliation, and finance's approval of adjustments and ledger entries. Governance features including row-level security, column masking, RBAC, ABAC, and shared business semantics keep access and terminology consistent, while Lakebase records corrections in auditable, timestamped tables. The result described is a closed-loop process in which strategy, risk, and accounting use the same governed data rather than reconciling separate systems.
Zalando describes how it is building a unified data foundation for analytics and AI across a microservices-driven ecosystem serving more than 50 million active customers and 7,000 brands and partners. Its architecture combines private domain-owned catalogs for autonomous development with a centrally governed shared catalog that exposes cross-team data through Dynamic Views, enabling identity-based policies, GDPR checks, column classification, and auditability. Databricks Metric Views provide a governed semantic layer where business logic and KPI definitions are centralized and reusable across SQL, dashboards, notebooks, and AI agents. Genie uses those metrics plus enriched metadata to answer natural-language questions more reliably, although poorly curated or unaggregated data can increase costs; Zalando plans broader enterprise adoption through curated spaces and additional orchestration.
Enterprise AI needs a way to assemble the smallest sufficient set of current, relevant, and authorized evidence for each agent turn instead of sending an entire data estate to a model. Salesforce presents Data 360 as a shared Trusted Context runtime spanning Salesforce, external platforms such as Databricks and Snowflake, operational systems, documents, knowledge bases, and live signals. Its progressive pipeline connects data through ingestion and Zero Copy, then harmonizes, unifies, models, retrieves, reconciles, governs, and compiles evidence into a token-fit Context Pack with citations and action contracts. The Agent Context Engine also returns typed interaction traces, promoting only approved facts into durable memory and validated outcomes into learnings. Q2 FY27 production usage is reported at 104.3 trillion connected records, including 81.5 trillion Zero Copy rows, while proposed benchmarking covers correctness, authorization, provenance, freshness, latency, cost, and memory continuity.
Health plan finance teams can see when medical loss ratio (MLR) moves above budget, but the variance alone does not explain which business areas, costs, populations, or providers caused it. Conversational AI is presented as a way for finance leaders to ask follow-up questions across claims, membership, provider, contract, clinical, quality, risk, and financial data without repeatedly commissioning reports. Its usefulness depends on explicit payer context: MLR combines premiums with medical-care components such as pharmacy rebates, IBNR, recoveries, risk adjustment transfers, and provider settlements, while cohort definitions and metric logic vary. Abacus supplies normalized, connected healthcare data and payer-specific business context; Databricks provides governed enterprise data and AI capabilities for conversational analytical experiences. Together, they are described as creating a reusable foundation for tracing executive signals into operational questions across MLR and other payer domains, while leaving action judgment to the organization.
Anthropic and Snowflake announced a multi-year, $200 million partnership that brings Claude models to Snowflake’s more than 12,600 customers across Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Azure. The agreement also includes a joint go-to-market initiative for deploying AI agents in large enterprises. Claude can analyze structured and unstructured data through Snowflake Cortex AI, while Snowflake Intelligence and Cortex Agents support natural-language questions, multimodal SQL-based analysis, and custom multi-agent solutions under Snowflake Horizon Catalog governance. Snowflake’s internal benchmarks report greater than 90% accuracy on complex text-to-SQL tasks, and examples include Simon Data and Intercom. The partnership is positioned to help regulated industries move AI agents from pilots to production within governed data environments.
Apache Iceberg’s REST Catalog additions—read restrictions and catalog labels—address different cross-platform governance problems: delegating enforcement to trusted engines and carrying governance context across federated catalogs. Read restrictions let a catalog evaluate policy for a principal and return bounded column-projection actions and row-filter expressions that a trusted engine must enforce, while leaving trust establishment outside the protocol. Catalog labels exchange opaque key-value metadata at table and column level, allowing a consuming catalog to map classifications locally, preserve native policy expressiveness, and maintain its own audit trail. Together with centralized enforcement through Cross-Engine ABAC, they provide distinct options for direct access and catalog federation, although read restrictions cannot express policies outside their standardized vocabulary.
Embedding a Databricks AI/BI Dashboard in a customer-facing application is straightforward; authorization determines which rows each viewer may see. The pattern supports external partners and internal Databricks teams with one dashboard and shared dataset, while applying different regional access and masking rules. An entitlements table stores partner IDs and IdP-synchronized group names alongside regions and PII-mask flags; a secured view filters on the signed __aibi_external_value from the embed token. Direct SQL access is separately protected by Unity Catalog row filters and column masks, which evaluate the querying user's identity and groups. The design provides a default-deny, pooled logical multi-tenancy model that centralizes rules while covering both application and workspace query paths.
Salesforce Health Insights addresses enterprise org health monitoring by combining configuration metadata with runtime telemetry to identify and prioritize remediation. The team evolved a static report into an in-app experience with durable finding dispositions, approximately 400+ signals across domains, and standards informed by Well-Architected principles and implementation anti-patterns. A shared harmonization layer reconciles fragmented datasets, connector patterns link runtime activity to metadata components, and the system preserves data origin, freshness, and access requirements while keeping deterministic findings distinct from externally supplied context. Prioritization combines findings with request volume, application and database CPU, call behavior, and peak-business-hour traffic to rank next best actions by criticality and effort. A headless JSON schema supports interfaces including an MCP connection, while the deterministic core is intended to produce dependable recommendations and extend access beyond the primary interface.
LTAP (Lake Transactional/Analytical Processing) addresses the long-standing separation between operational row-oriented databases and analytical column-oriented systems, which traditionally required pipelines and copied data. It uses Lakebase’s stateless, ephemeral compute decoupled from durable lake storage to expose live operational data for analytical queries without moving it or loading the transaction-serving system. The storage layer preserves Postgres data types and physical representation while writing data into Parquet, with a hot row-format tier for operational access and a cooler columnar tier for analytical reads; Apache Spark and SQL can read that representation directly. By separating serverless operational and analytical compute, LTAP allows each workload to use specialized engines and independent capacity, while a unified catalog applies governance across both. The stated motivation is agentic workloads such as fraud detection, where stale copies miss fast events and uncontrolled analytical queries can degrade transactions.
Marketing teams often have data across campaign, customer, attribution, and pipeline systems but still depend on analysts to answer questions quickly and consistently. Databricks Genie One is presented as a data-smart AI coworker that uses governed business data, organizational context, metric definitions, campaign structures, reporting rules, and access permissions to support marketing analysis. The five described uses cover channel and campaign performance, funnel attribution, customer insights, omnichannel planning, and mobile business reporting, with natural-language follow-ups, reusable skills or Genie Agents, scheduled analyses, dashboards, and shareable outputs. Examples include tracing sourced pipeline changes, identifying declining enterprise-account usage, and finding territories with weakening repeat purchases. The source recommends starting with repeatable, cross-source workflows and preparing data sources, attribution models, fiscal-calendar logic, and team terminology in Databricks; marketers retain responsibility for decisions and campaign plans.
Enterprise data products are presented as managed assets that deliver the right data at the right time, with appropriate quality, format, and usability for users and applications. Their value depends on clear ownership, business-aware data product owners, and five trustworthiness characteristics: quality and observability, semantic consistency, privacy, security, and discoverability. The proposed lifecycle runs from inception and design through creation, publication, operation and governance, consumption, and retirement, with data contracts, quality metrics, testing, versioning, access management, and lineage supporting those stages. For Databricks implementations, the guidance recommends Delta Live Tables expectations and constraints, staged development, testing and production environments, automated monitoring, and Unity Catalog for discovery, governance, lineage, auditing, and access controls.
The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.
Genie Ontology is presented as a way to give enterprise AI agents shared business context beyond a semantic model, including definitions, relationships, business rules, authoritative sources, and permissions. It combines Unity Catalog Semantics—Metric Views, Pages, and Domains—with context inferred from governed tables, queries, dashboards, notebooks, and other supported assets. The guidance recommends six progressive layers, beginning with clean gold data and resolved golden records, then metadata, semantic modeling, enterprise context, governance, and evaluation. Critical implementation details include declaring informational primary and foreign keys, defining canonical measures in Metric Views, adding synonyms and example queries, and using permissions plus human-reviewed automation. Rather than waiting for complete coverage, it advises starting with one high-value domain and metric, then using feedback, telemetry, benchmarks, and drift reviews to strengthen trust over time.
Public sector software must satisfy data protection, security, accessibility, procurement, supply-chain, and infrastructure requirements while protecting sensitive citizen data. Compliance failures can expose organizations to breaches, penalties, legal costs, contract disputes, and reputational damage. The brief organizes common risks around secure development, contractual and procurement obligations, audit evidence, long-term supportability, and IT governance, noting examples such as hardcoded credentials, weak cryptography, vulnerable dependencies, license conflicts, unsupported libraries, and unchecked quality issues. It recommends embedding automated SAST, secret detection, cryptography and dependency scans, license checks, quality gates, technical debt metrics, freshness tracking, testing, and time-stamped reports into IDE and CI/CD workflows, plus self-hosted or air-gapped deployment where policy requires it. These controls are presented as ways to catch problems before production, create audit trails, support handovers, and maintain compliance over long software lifecycles.
Enterprise intelligence requires more than the plausible answers produced by general-purpose LLMs: it needs accurate insights from vast real-world brand data, supported by statistics, governance, and domain expertise. LLMs can hallucinate, miss context, produce inconsistent answers, and rely on limited, non-randomly sampled sources, making them unsuitable as the sole basis for high-stakes decisions. The article argues for matching technologies to tasks, using statistical models for quantitative problems such as social and online-news Reach, specialized classifiers for other tasks, and generative AI to synthesize model outputs. At enterprise scale, crisis and reputation use cases also demand near-real-time speed, consistency, predictability, explainability, auditability, monitoring, and bias testing. LLMs still have a role, including as orchestrators grounded in high-quality data, workflows, specialized models, and rigorous evaluation; the proposed future combines varied models rather than choosing between LLMs and traditional AI.
Grab describes analytics as central to business decisions and presents AI agents as a way to democratise access to trustworthy answers. Over two years, increased model capability has enabled agents to prepare data, write and run queries, analyse opportunities, design experiments, interpret results, and draft commentary in minutes rather than a week. The proposed ladder ranges from AI-assisted work through human-planned and agent-planned workflows to end-to-end autonomy, while people retain accountability for framing, definitions, causal interpretation, decisions, and stakeholder relationships. Grab describes Spartan, which routes natural-language Slack requests across more than 50 skills and 120 analysis frameworks, plus Scarlet for pipeline failure triage and recovery. The programme also maintains agent context through ContextIQ and reports lower analyst intervention, a roughly 33% reduction in cycle times, and greater self-serve resolution in its support channel.
Data governance architecture is presented as a blueprint for aligning policies, roles, processes, and technologies with business outcomes. It defines objectives including consistent data definitions, data integrity, layered access controls, and secure self-service analytics, while assigning responsibilities across executives, architects, engineers, analysts, managers, and compliance teams. Core principles are accountability, transparency, consistency, and stewardship, supported by federated ownership through councils, data owners, and embedded stewards. The discussion compares DAMA-DMBOK, TOGAF, and Zachman according to organizational scale, regulatory context, and architecture maturity, and describes modern patterns including lakehouse, data mesh, and data fabric. It concludes that effective programs require executive sponsorship, documented roles, measurable quality controls, iterative implementation, and sustained change management.
The guide presents data warehouse design as a business-aligned process for building, modernizing, or scaling analytics infrastructure, rather than merely storing data. It recommends defining analytics use cases and stakeholder needs first, then organizing a three-tier architecture of source, storage, and semantic output layers; cloud designs can decouple compute and storage and use open formats. A Bronze–Silver–Gold medallion flow preserves raw lineage, applies cleansing and deduplication, and produces consumption-ready dimensional models, while retention and archival policies control sprawl. For modeling, it favors star schemas for user-facing BI, uses snowflake normalization when redundancy is material, and stresses explicit fact-table granularity, domain-owned data marts, and workload-specific refresh cadences. Governance and operations include Unity Catalog, access controls, masking, lineage, multi-region deployment, disaster recovery, and CI/CD, followed by phased rollout through high-value domains.
Databricks announces a broad set of Apache Iceberg capabilities in Unity Catalog, spanning General Availability, previews, and beta releases. Managed Iceberg is GA, supporting table creation, reads, writes, optimization, governance, and sharing, while Iceberg v3 adds deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables. Unity Catalog also federates external catalogs, vends credentials, shares live data with Iceberg REST-compatible clients through Delta Sharing, and applies attribute-based access control during server-side scan planning for supported external engines. These capabilities are presented as a unified approach to open APIs, cross-engine governance, zero-copy sharing, and production performance without copying data. The post also outlines Iceberg v4 and a proposal for Delta 5.0 to adopt an adaptive metadata tree structure.
Healthcare organizations often keep a FHIR server, analytics warehouse, and ETL pipelines separate, replicating clinical data and fragmenting governance. The proposed architecture standardizes incoming HL7v2, C-CDA, X12, and proprietary data into FHIR using Health Samurai converters, terminology normalization, MDM/MPI deduplication, and Implementation Guide validation. Aidbox, Health Samurai's FHIR server and database, runs on Databricks Lakebase, while Moonlink synchronizes operational and analytical formats without ETL. This exposes one governed dataset through Spark, SQL, ML, AI/BI, FHIR API, SMART on FHIR, and SQL on FHIR ViewDefinitions. The architecture supports use cases including EHR optimization, value-based care, member engagement, and compliance capabilities, with insights connected to clinician and billing workflows through SMART on FHIR and CDS Hooks.
The best engineering reads, once a week.
