# Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

[Grab](https://yomu.fyi/company/grab) · Yuanzhe Liu · Nov 26, 2025

## Summary

Kafka streams often suffer from syntactic and semantic data quality issues that propagate undetected to downstream consumers without real-time validation. Grab addressed this challenge by developing a standardized contract testing and observability framework within its Coban platform. Stakeholders define schema rules and field-level semantic validations, which can be recommended using large language models and anonymized sample data. A transformation engine converts these contracts into inverse SQL queries executed continuously by a FlinkSQL Test Runner on a dedicated consumer group. Problematic records are published to an alert topic, archived to AWS S3, and surfaced via Slack notifications and UI field-highlighting across more than 100 critical Kafka topics.

## Takeaways

- The Coban platform converts human-readable semantic test rules into inverse SQL queries executed by FlinkSQL to capture invalid and schema-mismatched data.
- Large language models assist users in generating field-level semantic test rules by analyzing Kafka stream schemas and anonymized sample data.
- The Test Runner operates on an independent Kafka consumer group and publishes enriched data quality events to both AWS S3 and a dedicated Kafka topic for alerting via the Genchi platform.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/real-time-data-quality-monitoring)