Improving Hugo stability and addressing oncall challenges through automation

Summary

Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.

Takeaways

• Hugo accelerates root-cause analysis by directly gathering computation engine and platform signals instead of parsing massive Airflow or Spark executor logs.
• Diagnostic parallelism scales dynamically during error spikes, using thousands of concurrent processes for failure signals and Kafka partition-based scaling for SLA and data completeness alerts.
• Pipeline health metadata is converted to a unified specification consumed by Kinabalu, automating incident management across Slack, Jira, Splunk on-call, and Datadog.

Related reading

• Grab · Oct 30, 2025

  Machine-learning predictive autoscaling for Flink

  Managing resource allocations for expanding Apache Flink stream-processing workloads often leads to over-provisioning or unstable reactive scaling. Reactive autoscaling with Kubernetes Horizontal Pod Autoscaling triggers severe restart spikes because pipelines must reprocess backlogged records from checkpoints, inducing feedback loops and scaling fluctuations. To prevent these spirals, a predictive vertical autoscaling system uses upstream Kafka message throughput as an independent workload metric. The architecture forecasts future workload using time-series models and maps the projected throughput to required TaskManager CPU allocations through a trained regression model. A custom controller vertically scales TaskManager CPU resources before traffic changes arrive, avoiding restart-induced latency spikes.

  Minh Nhat Nguyen

• Grab · Jan 21, 2026

  Docker lazy loading at Grab: Accelerating container startup times

  Grab addressed slow cold starts and auto-scaling bottlenecks caused by large container images across data platforms like Airflow and Spark Connect. To eliminate the requirement of downloading complete images before launching, the team evaluated Docker lazy loading via remote snapshotters using eStargz and Seekable OCI (SOCI). Unlike eStargz, which modifies image layers and increased application startup delays during benchmarks, SOCI stores index metadata separately as OCI Artifacts without altering image digests. In production on Amazon EKS, tuning SOCI concurrency and chunk parameters reduced fresh-node image download times by 60%, ultimately delivering a 30% to 40% reduction in P95 startup times.

  Huong Vuong

• Grab · Nov 21, 2025

  SpellVault’s evolution: Beyond LLM apps, towards the agentic future

  Grab developed SpellVault as an internal no-code platform to democratize the creation of AI applications backed by Retrieval-Augmented Generation (RAG) and plugin integrations. To advance beyond static retrieval and linear input-output processing, the platform transitioned from its legacy executor to a graph-based execution model supporting branching, looping, and ReAct agent patterns. Capabilities like Python code execution and internal repository searching were unbundled from the prompt builder and consolidated alongside user plugins into unified Native and Community Built Tools. The platform also introduced a drag-and-drop deterministic workflow designer, automated task scheduling, and support for the Model Context Protocol (MCP).

  Felix Haryanto Lie

• Grab · Jul 24, 2025

  The evolution of Grab's machine learning feature store

  Grab redesigned its initial machine learning feature store, Amphawa, to address high-dimensional data, complex entity retrieval, and versioning challenges during feature updates. The new architecture adopts a feature-table model where data scientists output Parquet datasets to Amazon S3 using Spark, which are then atomically ingested into Amazon Aurora PostgreSQL via a reverse ETL workflow. To prevent noisy-neighbor contention and optimize infrastructure costs, the platform utilizes Aurora's distributed storage to separate reads from writes. Grab pairs Aurora Serverless on writer nodes to scale up during daily batch ingestion with Provisioned instances on read replicas for steady serving traffic.

  Daniel Tai