# Modernising Grab’s model serving platform with NVIDIA Triton Inference Server

[Grab](https://yomu.fyi/company/grab) · Daniel Tai · Oct 21, 2025

## Summary

Grab's machine learning serving platform, Catwalk, experienced mounting technical debt, elevated latency, and rising costs from maintaining disparate inference engines for various frameworks. To address these limitations, the team adopted NVIDIA Triton Inference Server to establish a unified engine, starting with an ONNX migration. They built a custom Triton manager component featuring a proxy layer to translate legacy API requests alongside a server manager that handles model downloads, verification, configuration, and health checks. Within ten days, over half of online deployments migrated seamlessly without requiring client code modifications. The transition reduced p90 latency from 120ms to 20ms on large transformer models and cut average infrastructure spend by approximately 20 percent across evaluated services.

## Takeaways

- Setting the ONNX Runtime intra-op thread count to match the physical CPU core count provided a high-impact performance boost without requiring per-model micro-optimizations.
- A dedicated Triton proxy translated older API requests to preserve backward compatibility, allowing internal teams to adopt the new engine without changing their application code.
- Early rollout across 11 production machine learning services resulted in an average infrastructure cost reduction of roughly 20% over a 14-day comparison period.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Migrations](https://yomu.fyi/topic/migration), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://engineering.grab.com/modernising-grab-model-serving-platform)