Loading…
Topic
80 posts about Data Governance. Every summary links to the original.
See the idea
Antimicrobial resistance surveillance is a layered data architecture: raw reads and demand signals flow in, get normalized into governed tables, and answer plain-language queries. The diagram maps each boundary.
BYOK lets a system run on assets you supply — your model credential or your private knowledge — instead of the vendor's defaults. This walkthrough traces six steps from registration to an audited response.
Business Intelligence turns operational data into decisions, but every speed-up trades against trust. This diagram maps the benefit, cost, and condition behind governed metrics, shared context, and provenance.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
GrabMart transaction data could supplement clinical surveillance because over-the-counter demand sometimes shifts before confirmed case reports. The concept aggregates eligible purchases, maps SKUs to broad symptom classes, and compares demand series against seasonal and historical baselines to produce an Early Outbreak Detection Index. Public health experts would review that aggregate signal alongside independent evidence, and the method excludes identity linkage and individual-level health inference. Statistical evaluation would need sensitivity, false-alert rate, calibration, geographic stability, and retrospective testing against approved public health data, with ongoing monitoring for catalogue changes, promotions, and shifting purchasing behaviour. Privacy relies on data minimisation, coarse geography, aggregate time windows, minimum-count thresholds, and possibly differential privacy under an approved budget; the authors state this is early-stage planning only, with no deployment, measured performance, or integration claimed.
Notion built Data Scout, an internal data agent connecting Snowflake and Notion, letting employees query governed data in plain language and cutting request turnaround from days to minutes. Within two months, weekly data team requests fell by about 80 percent, and roughly half of all employees now use it. The agent enforces Snowflake permissions per user and uses Notion business context, enabling teams such as Sales, Engineering, and Customer Experience to ask questions about account adoption, onboarding drop-off, or support volume using shared definitions. Using MCP, teams built Data Scout Reporter for scheduled reports and domain-specific agents that reuse the same context. Notion learned correct SQL could still answer the wrong question due to four failure modes, so they built governed access, curated context, instructions, query tracing, and a review process where change evidence is not approval.
Buy-side asset management finance departments face growing complexity from AI agents influencing valuation, redemption, and reconciliation workflows, while protecting net fee margin and NAV integrity remains their core mission. Databricks Genie One is presented as a data-smart AI coworker grounded in Genie Ontology, which captures business meaning and keeps definitions current as strategies and fee arrangements shift. Behind the assistant, Genie Agents automate overnight fund operations: ingesting custodian files, pricing positions, reconciling records, and producing a preliminary NAV for review. Every figure is traceable to its source, permissioned by role, and provable to investors or regulators, enabling finance teams to see forming risks early. The system does not make investment or valuation decisions; a human remains in the loop for every action.
The Apache Iceberg community has adopted two additions to the Iceberg REST Catalog — read restrictions and catalog labels — each addressing distinct governance challenges. Read restrictions standardize delegated enforcement: a catalog evaluates policy for a principal and returns column-projection actions and row-filter expressions from a bounded vocabulary for a trusted engine to enforce. Catalog labels enable portable governance by allowing catalogs to exchange opaque key-value metadata at table and column level, so a consuming catalog applies native policies without per-user calls to the source. Read restrictions suit engine-to-catalog access where the source catalog trusts the engine and policies fit the standard vocabulary, whereas catalog labels fit federation between heterogeneous catalogs needing scalable governance and independent audit trails. These building blocks, along with centralized enforcement via cross-engine ABAC, advance the vision of unified governance across the open lakehouse.
As agentic applications move from answering questions to taking actions, enterprises risk AI sprawl from duplicated integrations, inconsistent policies, and fragmented context. Databricks asserts scaling requires three pillars: choice across models and tools, context grounded in governed enterprise data and business semantics, and control through consistent permissions, policies, observability, and cost management. Its platform provides Agent Bricks for unified agent fleet building and governance, Omnigent as a common layer above agent harnesses, and Unity Gateway as a centralized control plane for models, agents, and tools. Context is shared via Unity Catalog for data and AI asset governance and Genie Ontology for common business concepts, letting teams reuse governed context across applications. Control uses Unity Gateway for access policies, budgets, and traces and Databricks Sandbox for isolated execution, with MLflow enabling evaluation and workflow inspection to maintain reliability as agents scale.
The post examines why business intelligence (BI) dashboards lose user trust, even when data is accurate, focusing on how inconsistent metric definitions and opaque data sources cause users to doubt results. Richard Tomlinson argues that AI tools accelerate chart creation but also amplify trust debt if underlying data governance and semantics are weak, since AI can generate many inconsistent charts quickly. He recommends governing data and meaning, preserving provenance, and designing for inspection so visualizations invite investigation rather than presenting unexplained answers. Tomlinson stresses that dashboards should be self-explanatory, traceable, interactive, and context-aware, and that the critical metric for modern BI is ‘time to trust’ not just time to answer. Self-service fails when verifying an AI-generated answer takes longer than asking an analyst, so good visualization must enable inspection and challenge.
Unity Catalog managed tables let you control where your data physically resides while keeping it in cloud storage you own, such as S3, ADLS, or GCS, unlike platforms that store tables in provider-controlled storage. You set managed storage locations at the metastore, catalog, or schema level, with the most specific level winning, and can change where new tables land using ALTER CATALOG or ALTER SCHEMA commands. Managed tables use open formats including Iceberg and Delta, and external engines such as Apache Spark, Flink, and Trino can read and write through Iceberg REST Catalog and Unity Catalog open APIs. When converting an external table to managed, Databricks copies the data and transaction log into the managed storage location you have set. This approach enables a single source of truth across analytics and AI workloads while satisfying compliance, cost allocation, and data residency requirements.
Agentic AI transforms the database from a passive record store into an active control point that serves real-time context and enforces policy at the moment an agent acts. According to IDC’s April 2026 survey of 715 organizations, 30% already run agentic AI in production and another 50% have established spending plans. Real-time data delivery is now baseline: 64.6% of enterprise use cases require real-time or near-real-time delivery, and 91.2% of streaming scenarios need end-to-end latency under 200 milliseconds. IDC identifies four requirements for agentic AI: a real-time operational foundation for unstructured data, elastic scale for bursty agent traffic, accurate always-current retrieval via integrated vector databases, and a governed runtime with durable memory and identity-bound policy. By 2029, IDC predicts 60% of enterprise data platforms will unify transactional and analytical workloads in hybrid processing architectures to power agentic AI.
Manufacturing data is fragmented across systems such as MES, SCADA, QMS, ERP, and PLM, so answering a cross-stage question like whether a scrap spike stems from a supplier lot, machine setup, or process drift requires manual tickets, exports, and reconciliation. Databricks connects the product value chain by pooling or federating these sources using Lakehouse Federation and zero-copy Open Sharing, with Unity Catalog governing both mirrored and federated data under a single permission model and lineage. Lakeflow orchestrates the refinement of raw data into trusted Bronze, Silver, and Gold tables, while Genie and Agent Bricks enable natural-language queries grounded in governed business semantics. A quality engineer can thus ask about a defect’s history across plants and get answers that join MES, supplier, and QMS records without migrating every system. The result is that traceability becomes a query, transforming weeks of manual investigation into minutes of analysis across the entire value chain.
S&P Global Energy aimed to make its structured data estate—spanning LNG, Chemicals, Crude Oil, and other commodities—conversationally accessible via AI agents. The team evaluated approaches and chose Databricks Genie Agents, where subject matter experts curate one agent per dataset group without writing code. Each Genie Agent is automatically exposed as a managed MCP server, providing a governed, zero-deployment integration point. A FastMCP-based proxy composes multiple Genie MCP servers into composite endpoints for cross-domain questions. This architecture shortened time to market from months to days, with governance via Unity Catalog and an open MCP standard enabling both internal and external consumers.
Business leaders often have access to plenty of data but still cannot get a reliable answer to a simple question like why net revenue declined 8% at their largest account last week. The answer may span sales, finance, promotions, inventory, and account data, with each source potentially correct in isolation yet different in definitions, detail, relationships, and authority. Without the right business context, general-purpose agents cannot reliably determine what metrics mean, which sources are authoritative, how data should connect, or what each user is allowed to see. Genie One MCP gives governed context to any MCP-compatible AI agent, connecting assistants like ChatGPT, Claude, Microsoft Copilot, and coding agents to Genie One. This lets them answer business questions using approved definitions, trusted data relationships, and permission-aware access controls, with traceable, permission-aware answers grounded in governed business context.
AI analytics applies AI and machine learning to data analysis, enabling natural-language queries and automated insights without manual query-building, shifting from retrospective dashboards toward agentic investigations. Unlike traditional BI, which requires users to navigate predefined dashboards and filters, AI analytics systems understand intent, determine relevant data and business context, and can conduct multi-step investigations on behalf of the user. A key challenge is that AI analytics only works when grounded in governed data: without shared metric definitions, authoritative sources, and permission-aware execution, the system produces untrustworthy results. Trust in AI analytics is a system property, not a model property; it requires governed data, shared business context, permission enforcement, and verifiability of queries and calculations. Better AI models do not eliminate the need for business context; as AI becomes more autonomous, semantic definitions and governance become more critical.
India’s Digital Personal Data Protection Act (DPDPA), made law in 2023 and enforced in 2025, requires organizations to implement technical controls for consent, access, security, retention, and deletion of personal data. Traditional relational architectures often fragment customer information across multiple tables and services, making it difficult to track consent or respond to correction and deletion requests. MongoDB’s capabilities, including its flexible document model, change streams, queryable encryption, and search, can help consolidate privacy-relevant data and context. The document model allows storing consent state, personal data, and security controls together, reducing fragmentation and simplifying auditability. MongoDB Atlas and Enterprise Advanced offer deployment options for fully managed or self-hosted environments to support privacy-by-design architectures.
Capital-markets firms are revisiting the trade lifecycle as data volumes grow, AI moves toward production, settlement cycles shorten, and scrutiny increases. Fragmented data across research, trading, risk, operations, and surveillance can slow decision-making, complicate execution analysis, limit risk visibility, and hinder reconstruction of events. The proposed response starts with decisions such as execution-cost divergence, shock impacts, and exception or settlement-fail rates, then identifies the data domains needed for a few high-value workflows rather than migrating everything at once. It emphasizes a governed, reusable foundation connecting real-time and historical data, analytics, and AI, with access controls, lineage, traceability, and evaluation. Databricks components named include Unity Catalog for governance, discovery, and lineage, Agent Bricks for domain-specific agents, and AI/BI for governed exploration; the stated conclusion is that this operating-model approach can support faster decisions, resilience, and scalable use cases.
Databricks says it was recognized as a Customers’ Choice in Gartner Peer Insights’ Voice of the Customer for Analytics and Business Intelligence Platforms, based on verified reviews from customers using platforms in production. The post reports a 4.8 out of 5 overall rating and 94% willingness to recommend, while noting that the designation reflects customer adoption and overall experience meeting or exceeding market averages. It presents Databricks AI/BI, AI/BI Dashboards, Genie Spaces, and Genie as a unified architecture combining governed live data, AI, business intelligence, and Unity Catalog controls without moving or duplicating datasets. The described use cases include conversational exploration, dashboard and visualization development, consolidating fragmented reporting, and expanding governed analytics access, with the company positioning these capabilities as a foundation for agentic analytics.
Memory scaling is proposed as an agent’s performance improving when it accumulates past conversations, feedback, interaction trajectories, and business context in persistent external memory. Unlike model-parameter or inference-time scaling, it relies on selective retrieval to supply relevant information without filling long context windows. In MemAlign experiments on Databricks Genie Spaces, adding labeled examples raised test scores from near zero to 70%, about 5% above an expert-curated baseline, while average reasoning steps fell from about 20 to 5; filtered user logs produced similarly strong gains, exceeding 50% after 62 records. Production use requires storage and retrieval for episodic and semantic memories with organizational and user-level scoping, plus controls that preserve permissions and sensitivity labels.
Business analytics tools have evolved beyond static reporting as organizations seek explanations, forecasts, and immediate actions. This guide categorizes visualization and dashboard platforms, self-service analytics, statistical tools, spreadsheets, and SQL-based query tools, then examines how AI adds natural-language querying, anomaly detection, summaries, and forecasting. It argues that inconsistent, duplicated, or differently governed data can undermine even sophisticated dashboards. Lakehouse architecture combines data-lake scalability with warehouse governance, performance, and SQL compatibility, enabling tools such as Tableau, Power BI, and Looker to use broader, fresher datasets alongside analytics and AI workloads. Reported examples include 94% faster BI queries at Anker Innovations, 98% faster query performance at AnyClip, and consolidation across 120+ global analysts at JLL.
Databricks announces the general availability of Unity Catalog Business Semantics, a unified semantic foundation for consistent business definitions across BI dashboards, notebooks, SQL applications, and AI agents. Built into Unity Catalog, it applies existing governance, security, lineage, and access controls to centrally defined Metric Views, whose metadata includes display names, formats, and synonyms. Metric Views are authored declaratively in SQL, compiled and executed deterministically at query time, and can use materializations with automatic pre-aggregation, incremental refresh, intelligent query rewriting, and transparent routing; materialization remains in Preview. The core Metric View implementation is being open sourced in Apache Spark OSS for an upcoming release, with Unity Catalog OSS v0.5 support coming soon, while a point-and-click authoring UI is in Public Preview.
Agentic analytics increases the need for trustworthy semantics and governance because AI systems must interpret business definitions, trace outputs, and support follow-up questions beyond static dashboards. Nick Eayrs argues that fragmented metrics across BI tools, proprietary semantic layers, and duplicated pipelines erode trust, slow decisions, and increase engineering cost. The proposed foundation is an open, interoperable semantic layer where certified metrics such as revenue, churn, and customer lifetime value are explicitly defined, reusable, and accessible through SQL to BI, AI interfaces, notebooks, and agents. Organizations should establish governed data foundations and a catalog first, certify metrics next, then add AI and evaluate its outputs. Examples include NTT Docomo reducing manual log-analysis time from 66 hours to 6 hours and Net One Systems cutting support response time by 75 percent.
Databricks is introducing a Lakeflow Connect Free Tier to help teams ingest SaaS application and operational database data into the Databricks Data + AI Platform, addressing silos that can leave AI agents without complete enterprise context. Each workspace automatically receives 100 free DBUs per day for managed SaaS and database connectors only, with capacity advertised as up to 100 million records per workspace per day across eligible supported sources. Lakeflow Connect offers 30+ connectors, including SaaS services and databases, through a point-and-click UI and API, with Unity Catalog providing unified governance and end-to-end lineage. The offer uses standard pricing without volume-based premiums or row-based pricing, but actual capacity varies with workload characteristics, record sizes, and source types.
The best engineering reads, once a week.
