Ensuring data reliability and observability in risk systems

Summary

Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.

Takeaways

1. 1

   Custom Flink SQL table functions like JSONEXPLOAD deconstruct nested JSON streams into tabular formats suitable for 5-minute tumbling window aggregations.

2. 2

   Grouping Datadog counters by source stream and using Datadog Monitor Summary resolved visualization clutter caused by hundreds of distinct data points.

3. 3

   The integrated Flink, Datadog, and Slack pipeline reduced anomaly notification times from days or weeks down to within the same day or hour.