Modernising Grab’s model serving platform with NVIDIA Triton Inference Server

Summary

Catwalk, Grab’s machine learning model serving platform, accumulated technical debt by maintaining separate inference engines for ONNX, PyTorch, TensorFlow, and vLLM, contributing to higher latency, lower throughput, and rising costs. The team evaluated NVIDIA Triton Inference Server and began consolidating those engines around it, initially adopting ONNX. The Triton manager combines a server manager, which retrieves and verifies models, prepares configurations, launches Triton, and monitors health, with a proxy that translates legacy API requests without requiring client code changes. Benchmarks showed at least five times the traffic capacity for a transformer model; rollout reached over 50% of online deployments in 10 days, while some larger models saw p90 latency fall from 120 ms to 20 ms.

Takeaways

1. 1

   Triton handled at least 5 times the traffic of Catwalk’s existing ONNX server for a transformer-based model while maintaining excellent latency; batching provided additional performance improvements.

2. 2

   More than 50% of online model deployments migrated within 10 days, with backward-compatibility testing allowing most users to remain unaware of the transition.

3. 3

   Across 11 migrated production ML services, average infrastructure spend fell by approximately 20% over the 14 days after cut-over, while some individual models achieved over 90% cost reductions.