GrabShuguang XiangProblem & solution
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
Hugo pipeline failures can require user intervention when retries are insufficient. Users have limited visibility into issues, resolution steps are difficult to identify across extensive documentation, the on-call team faces ad-hoc requests and growing infrastructure dependencies, and DPIs may lack clear root-cause analysis, increasing data downtime.
The solution combines signal collection, automated diagnosis, an RCA table, asynchronous auto-resolution, a data health API, and the Data Health Workbench. Airflow callbacks and Genchi generate health signals; diagnosis uses signals instead of extensive log parsing and runs analysers in parallel, while custom handlers can apply fixes such as retrying jobs with backoff for transient errors.
The Signal module combines failure callbacks, SLA alerts, and data quality test failures to define a data lake table’s health status.
Diagnosis determines the failure root cause, responsible assignee, auto-resolution method, and manual resolution steps; parallelism varies by signal type to reduce latency during error surges.
The Data Health Workbench shows table health, assignee information, fix instructions, auto-resolution status, and a manual action button, with additional diagnosis details for administrators.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Airflow is a workflow orchestration platform that schedules, monitors, and manages data pipelines by defining tasks as code, executing them in dependency order, and tracking their success or failure.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
