# Improving Hugo stability and addressing oncall challenges through automation

[Grab](https://yomu.fyi/company/grab) · Shuguang Xiang · Mar 20, 2025

## Summary

Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.

## Takeaways

- Hugo accelerates root-cause analysis by directly gathering computation engine and platform signals instead of parsing massive Airflow or Spark executor logs.
- Diagnostic parallelism scales dynamically during error spikes, using thousands of concurrent processes for failure signals and Kafka partition-based scaling for SLA and data completeness alerts.
- Pipeline health metadata is converted to a unified specification consumed by Kinabalu, automating incident management across Slack, Jira, Splunk on-call, and Datadog.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Incident Response](https://yomu.fyi/topic/incident-response), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/improving-hugo-stability)