---
title: "Data Quality"
description: "12 posts about Data Quality, summarised, each linking to the original."
---

# Data Quality
> 12 posts about Data Quality, summarised, each linking to the original.

## Articles

### [Data Governance Architecture: A Complete Blueprint for Modern Organizations](https://yomu.fyi/post/data-governance-architecture-a-complete-blueprint-for-modern-organizat.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 2, 2026

Data governance architecture is presented as a blueprint for aligning policies, roles, processes, and technologies with business outcomes. It defines objectives including consistent data definitions, data integrity, layered access controls, and secure self-service analytics, while assigning responsibilities across executives, architects, engineers, analysts, managers, and compliance teams. Core principles are accountability, transparency, consistency, and stewardship, supported by federated ownership through councils, data owners, and embedded stewards. The discussion compares DAMA-DMBOK, TOGAF, and Zachman according to organizational scale, regulatory context, and architecture maturity, and describes modern patterns including lakehouse, data mesh, and data fabric. It concludes that effective programs require executive sponsorship, documented roles, measurable quality controls, iterative implementation, and sustained change management.


### [AI readiness in telecommunications](https://yomu.fyi/post/ai-readiness-in-telecommunications.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Stephen Hage, Keerthi Josyula, Michael Zhang
- Published: May 26, 2026

Telecommunications companies are adopting AI for customer experience, network operations, and cost reduction, yet initiatives often stall before production because fragmented, ungoverned, semantically opaque data creates data debt. The post argues that AI readiness depends on a semantic layer unifying datasets and business definitions, governance, and catalog metadata across systems such as Oracle, Snowflake, Salesforce, ServiceNow, and Databricks. It presents Unity Catalog as the proposed foundation, using Delta Sharing, Lakeflow Connectors, and Lakehouse Federation to exchange, ingest, or query data without uniformly replicating it, while privilege-aware metadata and audit logging support compliance. Metric Views, lineage, tags, and glossaries give agents authoritative meanings for measures and terms such as revenue, ARPU, active user, and FTTH. The conclusion is that trustworthy operational AI requires a governed, unified data foundation and organizational commitment, not simply more capable models.


### [Automate Data & KPI Monitoring with SQL Alerts](https://yomu.fyi/post/automate-data-kpi-monitoring-with-sql-alerts.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Srilekha Dornadula
- Published: May 19, 2026

Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue\_pct\_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic.


### [PipelineIQ: Forward‑Looking Sales Intelligence That Drives Action](https://yomu.fyi/post/pipelineiq-forward-looking-sales-intelligence-that-drives-action.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sam Le Corre, Dael Williamson, Luis Herrera
- Published: May 15, 2026

PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai\_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.


### [Data quality is the AI strategy](https://yomu.fyi/post/data-quality-is-the-ai-strategy.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: May 13, 2026

NYU Langone Health’s AI strategy starts with data quality, arguing that healthcare AI cannot be reliable when source data is fragmented or inconsistent. The institution standardized on common transactional platforms, including one electronic health record and one ERP system, established authoritative data sources, and fixes data at the source rather than mapping it in the warehouse layer. Its Databricks-based unified data and AI platform, with Unity Catalog, supports clinicians, analysts, scientists, and corporate users across care, operations, and research, while real-time feeds power emergency-room decision-support models. Mherabi also describes a three-layer analytics model: structured visualizations, conversational tools such as Genie, and answers delivered in formats suited to the user. The stated conclusion is that upstream data discipline, governance, literacy, and adaptable platforms provide the foundation for trustworthy AI and timely clinical insight.


### [Machine Learning Solutions: A Complete Implementation Guide](https://yomu.fyi/post/machine-learning-solutions-a-complete-implementation-guide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026

The guide presents machine learning delivery as a lifecycle problem, arguing that failures arise more often from planning, scoping, communication, and expectations than from algorithmic limitations. It recommends assessing data readiness before development by inventorying sources, checking quality and coverage, planning labeling, and engineering features. Once a measurable business outcome is defined, teams should match the learning paradigm and model complexity to the problem, establish cross-validation and holdout or temporal validation, and estimate compute and storage early. Baselines, reproducible experiments, deployment monitoring, model versioning, CI/CD, and retraining workflows support the transition from prototype to production. The guide also calls for representative proof of concepts and A/B testing or control groups to attribute business impact before scaling.


### [Data Governance Platforms: Evaluation & Feature Guide](https://yomu.fyi/post/data-governance-platforms-evaluation-feature-guide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026

This guide presents a framework for evaluating data governance platforms for enterprise deployment, distinguishing governance—the policies, roles, and controls for data use—from data management, the operational execution of those policies. It recommends assessing metadata-centered capabilities including continuous data profiling, catalog search and enrichment, end-to-end lineage, RBAC and ABAC, sensitive-data detection, masking, audit trails, compliance reporting, and data-subject request workflows. Vendor assessment should also cover pre-built connectors, REST APIs and SDKs, near-real-time metadata synchronization, schema-drift handling, cross-cloud federation, usability, support, implementation timelines, and three-to-five-year total cost of ownership. The proposed decision process selects three leading candidates, pilots them with representative structured and unstructured datasets, defines quality, lineage, adoption, and exit metrics, and uses executive review before procurement; governance is framed as an ongoing program that expands with AI workloads and regulatory change.


### [Building an Enterprise Data Management Strategy](https://yomu.fyi/post/building-an-enterprise-data-management-strategy.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026

Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.


### [AI Data Transformation Guide for Data Engineers and Data Scientists](https://yomu.fyi/post/ai-data-transformation-guide-for-data-engineers-and-data-scientists.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026

AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.


### [What is operational analytics?](https://yomu.fyi/post/what-is-operational-analytics.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 15, 2026

Operational analytics uses real-time or near-real-time data to monitor day-to-day operations and support immediate decisions, unlike traditional analytics, which generally relies on batch data to explain past performance. Its workflow collects logs, events, clickstreams, telemetry, transactions, and other signals; streams them into a unified platform for cleaning, transformation, and enrichment; analyzes incoming data with rules, models, or anomaly detection; and sends results to dashboards, alerts, or operational applications. This can help teams detect issues earlier, reduce mean time to detect (MTTD) and mean time to respond (MTTR), improve forecasting, coordinate across departments, and act on inventory, customer, or system changes. The approach also requires integrating heterogeneous systems, maintaining data quality and schemas, embedding insights into existing workflows, and building reliable low-latency pipelines, with tools such as Lakeflow, Databricks SQL, and AI and machine learning capabilities presented as examples.


### [A Fall Internship at Ramp: Improving Reimbursements](https://yomu.fyi/post/a-fall-internship-at-ramp-improving-reimbursements.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Sat Arora
- Published: Apr 3, 2025

During a fall internship at Ramp, a backend software engineering intern worked on the reimbursements team and describes projects that improved reimbursement flows while providing full-stack experience. One early change clarified multi-currency mileage reimbursements by showing both the expense currency and the payout currency, while later work overhauled mileage support with live navigation storage, frequent-trip recommendations, commute deductions, and automatic e-receipts. The post also discusses using LLMs to predict merchant categories from receipt text, enhancing merchant matching, and adding monitoring and alerts for OCR categorization errors. The reflection emphasizes measuring feature usage and outcomes, writing and revising specifications before implementation, leaving reusable code, and using Ramp’s transparent documentation and collaborative support. The internship concluded with a return to the University of Waterloo planned for 2025.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025

Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.