# Ensuring data reliability and observability in risk systems

[Grab](https://yomu.fyi/company/grab) · Yi Ni Ong · Apr 23, 2024

**Type:** Problem & solution

## Summary

Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.

## Context

GrabDefence relies on large volumes of upstream data for real-time risk rules and models, meaning data discrepancies or missing data directly impact fraud detection and prevention.

## Approach / What changed

Standardizing transformations with Flink SQL to parse nested JSON via custom functions and aggregate streams over 5-minute tumbling windows, then pushing metrics to Datadog for stream-grouped anomaly detection and Slack alerting.

## Takeaways

- Custom Flink SQL table functions like JSONEXPLOAD deconstruct nested JSON streams into tabular formats suitable for 5-minute tumbling window aggregations.
- Grouping Datadog counters by source stream and using Datadog Monitor Summary resolved visualization clutter caused by hundreds of distinct data points.
- The integrated Flink, Datadog, and Slack pipeline reduced anomaly notification times from days or weeks down to within the same day or hour.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/data-observability)