Grab's service mesh evolution: From Consul to Istio

Summary

Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.

Takeaways

• Grab's legacy Consul setup relied on a fallback mechanism called Catcher, which added complexity and hindered advanced circuit breaking and retry policies.
• Instead of running a control plane inside each cluster, Grab deployed multiple control planes on dedicated Kubernetes clusters in active-active pairs for isolation and high availability.
• The Istio migration began in Q4 2024 with a GCP-to-AWS cross-cloud transition alongside parallel initiatives to migrate HTTP and gRPC traffic across meshes without downtime.

Related reading

• Grab · Jun 20, 2025

  Counter Service: How we rewrote it in Rust

  The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance.

  Jia Long Loh

• Grab · Oct 21, 2025

  Modernising Grab’s model serving platform with NVIDIA Triton Inference Server

  Grab's machine learning serving platform, Catwalk, experienced mounting technical debt, elevated latency, and rising costs from maintaining disparate inference engines for various frameworks. To address these limitations, the team adopted NVIDIA Triton Inference Server to establish a unified engine, starting with an ONNX migration. They built a custom Triton manager component featuring a proxy layer to translate legacy API requests alongside a server manager that handles model downloads, verification, configuration, and health checks. Within ten days, over half of online deployments migrated seamlessly without requiring client code modifications. The transition reduced p90 latency from 120ms to 20ms on large transformer models and cut average infrastructure spend by approximately 20 percent across evaluated services.

  Daniel Tai

• Grab · Nov 6, 2025

  Grab's Mac Cloud Exit supercharges macOS CI/CD

  Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings.

  Madushan Gamage

• Grab · May 19, 2025

  From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT

  Grab's machine learning platform team initially faced overwhelming volumes of repetitive user inquiries across their internal support channels. An initial attempt to automate answers using the open-source chatbot-ui framework and GPT-3.5-turbo failed to scale because the 8,000-token context limit could not accommodate extensive documentation, and embedding search proved inadequate. The project then pivoted to create an internal conversational AI platform called GrabGPT by wiring chatbot-ui with Google authentication and Grab's catwalk model-serving infrastructure. The resulting internal service rapidly expanded across the organization, providing auditable interactions, multi-model support across OpenAI, Claude, and Gemini, and private network routing to safeguard corporate data.

  Wenbo Wei