Loading…
Topic
19 posts about Data Quality. Every summary links to the original.
See the idea
Business Intelligence turns operational data into decisions, but every speed-up trades against trust. This diagram maps the benefit, cost, and condition behind governed metrics, shared context, and provenance.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Intercom’s VP of Design Emmet Connolly and Staff Product Designer Molly Mahar discuss how product designers must adapt their methods for AI, drawing from their experiences building Intercom’s products. They explain that designing for AI means moving from deterministic to probabilistic systems, where unpredictability is the norm and designers must design flexibly around surprises. Data quality is foundational for reliable AI, so designers should prioritize it; realistic prototyping using actual data helps capture essential user feedback and evaluate output effectiveness. Extensive testing is necessary to anticipate possible system failures; if live versions catch designers off guard, testing must be more robust. Building user trust requires consistent positive experiences and transparency about how the system works and its data, enabling designers to create AI experiences that make users feel comfortable and confident.
Enterprise data products are presented as managed assets that deliver the right data at the right time, with appropriate quality, format, and usability for users and applications. Their value depends on clear ownership, business-aware data product owners, and five trustworthiness characteristics: quality and observability, semantic consistency, privacy, security, and discoverability. The proposed lifecycle runs from inception and design through creation, publication, operation and governance, consumption, and retirement, with data contracts, quality metrics, testing, versioning, access management, and lineage supporting those stages. For Databricks implementations, the guidance recommends Delta Live Tables expectations and constraints, staged development, testing and production environments, automated monitoring, and Unity Catalog for discovery, governance, lineage, auditing, and access controls.
Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment.
Data governance architecture is presented as a blueprint for aligning policies, roles, processes, and technologies with business outcomes. It defines objectives including consistent data definitions, data integrity, layered access controls, and secure self-service analytics, while assigning responsibilities across executives, architects, engineers, analysts, managers, and compliance teams. Core principles are accountability, transparency, consistency, and stewardship, supported by federated ownership through councils, data owners, and embedded stewards. The discussion compares DAMA-DMBOK, TOGAF, and Zachman according to organizational scale, regulatory context, and architecture maturity, and describes modern patterns including lakehouse, data mesh, and data fabric. It concludes that effective programs require executive sponsorship, documented roles, measurable quality controls, iterative implementation, and sustained change management.
Telecommunications companies are adopting AI for customer experience, network operations, and cost reduction, yet initiatives often stall before production because fragmented, ungoverned, semantically opaque data creates data debt. The post argues that AI readiness depends on a semantic layer unifying datasets and business definitions, governance, and catalog metadata across systems such as Oracle, Snowflake, Salesforce, ServiceNow, and Databricks. It presents Unity Catalog as the proposed foundation, using Delta Sharing, Lakeflow Connectors, and Lakehouse Federation to exchange, ingest, or query data without uniformly replicating it, while privilege-aware metadata and audit logging support compliance. Metric Views, lineage, tags, and glossaries give agents authoritative meanings for measures and terms such as revenue, ARPU, active user, and FTTH. The conclusion is that trustworthy operational AI requires a governed, unified data foundation and organizational commitment, not simply more capable models.
Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue_pct_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic.
PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.
NYU Langone Health’s AI strategy starts with data quality, arguing that healthcare AI cannot be reliable when source data is fragmented or inconsistent. The institution standardized on common transactional platforms, including one electronic health record and one ERP system, established authoritative data sources, and fixes data at the source rather than mapping it in the warehouse layer. Its Databricks-based unified data and AI platform, with Unity Catalog, supports clinicians, analysts, scientists, and corporate users across care, operations, and research, while real-time feeds power emergency-room decision-support models. Mherabi also describes a three-layer analytics model: structured visualizations, conversational tools such as Genie, and answers delivered in formats suited to the user. The stated conclusion is that upstream data discipline, governance, literacy, and adaptable platforms provide the foundation for trustworthy AI and timely clinical insight.
Grab’s Signals Marketplace uses three foundational platforms to make data certification operational across its data mesh: Hubble, Genchi, and the Data Contract Registry. Hubble, built on DataHub, catalogs assets, ownership, documentation, lineage, contracts, and certification status, while its event-driven engine recalculates certification as metadata and upstream health change. Genchi provides self-service observability and tests for freshness, volume, schema, completeness, and business checks; the registry manages explicit expectations between producers and consumers. Together, these signals let Hubble classify assets as Uncertified, Certified, CertifiedPlus, or Revoked, giving consumers a workflow centered on certified data rather than hearsay. The post reports that P80 datasets, the tables accounting for 80% of queries, fell by over 58% in one year after the Signals Marketplace campaign began in 2024, and positions the foundation as support for AI agents and automated analytics.
The guide presents machine learning delivery as a lifecycle problem, arguing that failures arise more often from planning, scoping, communication, and expectations than from algorithmic limitations. It recommends assessing data readiness before development by inventorying sources, checking quality and coverage, planning labeling, and engineering features. Once a measurable business outcome is defined, teams should match the learning paradigm and model complexity to the problem, establish cross-validation and holdout or temporal validation, and estimate compute and storage early. Baselines, reproducible experiments, deployment monitoring, model versioning, CI/CD, and retraining workflows support the transition from prototype to production. The guide also calls for representative proof of concepts and A/B testing or control groups to attribute business impact before scaling.
This guide presents a framework for evaluating data governance platforms for enterprise deployment, distinguishing governance—the policies, roles, and controls for data use—from data management, the operational execution of those policies. It recommends assessing metadata-centered capabilities including continuous data profiling, catalog search and enrichment, end-to-end lineage, RBAC and ABAC, sensitive-data detection, masking, audit trails, compliance reporting, and data-subject request workflows. Vendor assessment should also cover pre-built connectors, REST APIs and SDKs, near-real-time metadata synchronization, schema-drift handling, cross-cloud federation, usability, support, implementation timelines, and three-to-five-year total cost of ownership. The proposed decision process selects three leading candidates, pilots them with representative structured and unstructured datasets, defines quality, lineage, adoption, and exit metrics, and uses executive review before procurement; governance is framed as an ongoing program that expands with AI workloads and regulatory change.
Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.
AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.
Operational analytics uses real-time or near-real-time data to monitor day-to-day operations and support immediate decisions, unlike traditional analytics, which generally relies on batch data to explain past performance. Its workflow collects logs, events, clickstreams, telemetry, transactions, and other signals; streams them into a unified platform for cleaning, transformation, and enrichment; analyzes incoming data with rules, models, or anomaly detection; and sends results to dashboards, alerts, or operational applications. This can help teams detect issues earlier, reduce mean time to detect (MTTD) and mean time to respond (MTTR), improve forecasting, coordinate across departments, and act on inventory, customer, or system changes. The approach also requires integrating heterogeneous systems, maintaining data quality and schemas, embedding insights into existing workflows, and building reliable low-latency pipelines, with tools such as Lakeflow, Databricks SQL, and AI and machine learning capabilities presented as examples.
Grab’s Analytics Data Warehouse team supports more than 1,000 monthly users and 15,000 tables, but repetitive questions about definitions, lineage, quality, and basic enhancements consumed about 40% of engineering time. It deployed a multi-agent AI system that routes Slack requests into enhancement or investigation pathways, using FastAPI and LangGraph for stateful, cyclical collaboration, Redis for caching and sessions, and PostgreSQL for persistent memory. Specialized agents query Trino, Hive, or Delta Lake, inspect GitLab code, check incidents and pipeline health, or prepare semi-automated code changes, while human reviewers approve production modifications. The system autonomously resolves most standard inquiries and some common enhancements, reducing resolution time by an order of magnitude and bringing simple answers to users within minutes. The resulting capacity shift reclaimed several FTEs worth of engineering bandwidth and moved hundreds of hours from reactive support to roadmap work.
Kafka stream data can fail syntactically when actual records do not match expected schemas, or semantically when fields violate agreed patterns, ranges, or constants. The described platform addresses this gap with user-defined data contracts covering schema agreements, field-level semantic rules, stream configuration, and ownership metadata. An LLM recommends semantic rules from schemas and anonymized samples, while a transformation engine converts them into Test Runner configurations and inverse SQL queries. A long-running FlinkSQL Test Runner reads Kafka with its own consumer group, publishes issue events and stores them in AWS S3; Genchi sends Slack alerts and Coban shows violating fields and sample records. Deployed across 100+ critical topics, the solution provides real-time detection, alerting, and visibility, with future work targeting producer tracing, cross-field tests, completeness, freshness, and DTO change detection.
Grab began its Signals Marketplace data mesh journey in 2024 because its expanding, varied data ecosystem had ownership gaps, duplicated pipelines, central Data Engineering bottlenecks, broken downstream dependencies, inconsistent definitions, and uneven practitioner expertise. The decentralised model treats data as a product owned by domain teams, with Business Data Owners and Technical Data Owners accountable for upkeep, documentation, usability, and SLOs. Certification formalises expectations through data contracts covering schema, freshness, completeness, retention, change notice, and communication, while automated Data Production Incidents flag breached reliability guarantees for investigation. Using the percentage of queries hitting certified assets as its north star metric, Grab reports 75% of queries targeting certified assets, a 400% year-over-year increase in deprecated tables, and a 58% reduction in P80 datasets.
During a fall internship at Ramp, a backend software engineering intern worked on the reimbursements team and describes projects that improved reimbursement flows while providing full-stack experience. One early change clarified multi-currency mileage reimbursements by showing both the expense currency and the payout currency, while later work overhauled mileage support with live navigation storage, frequent-trip recommendations, commute deductions, and automatic e-receipts. The post also discusses using LLMs to predict merchant categories from receipt text, enhancing merchant matching, and adding monitoring and alerts for OCR categorization errors. The reflection emphasizes measuring feature usage and outcomes, writing and revising specifications before implementation, leaving reusable code, and using Ramp’s transparent documentation and collaborative support. The internship concluded with a return to the University of Waterloo planned for 2025.
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
The best engineering reads, once a week.
