Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

Summary

Kafka stream data can fail syntactically when actual records do not match expected schemas, or semantically when fields violate agreed patterns, ranges, or constants. The described platform addresses this gap with user-defined data contracts covering schema agreements, field-level semantic rules, stream configuration, and ownership metadata. An LLM recommends semantic rules from schemas and anonymized samples, while a transformation engine converts them into Test Runner configurations and inverse SQL queries. A long-running FlinkSQL Test Runner reads Kafka with its own consumer group, publishes issue events and stores them in AWS S3; Genchi sends Slack alerts and Coban shows violating fields and sample records. Deployed across 100+ critical topics, the solution provides real-time detection, alerting, and visibility, with future work targeting producer tracing, cross-field tests, completeness, freshness, and DTO change detection.

Takeaways

1. 1

   Data contracts combine Kafka schemas with field-level semantic rules such as string patterns, number ranges, and constant values, plus stream ownership and alerting metadata.

2. 2

   FlinkSQL consumes the monitored topic with a separate consumer group and uses inverse SQL queries to capture syntactically incorrect or semantically invalid records without affecting other consumers.

3. 3

   The deployed solution monitors 100+ critical Kafka topics and shows sample bad records and violating fields; planned extensions include producer tracing, cross-field validation, completeness, freshness, and DTO change detection.