# Catwalk: Serving Machine Learning Models at Scale

[Grab](https://yomu.fyi/company/grab) · Nutdanai Phansooksai · Jul 2, 2019

**Type:** Problem & solution

## Summary

As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved\_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.

## Context

Teams at Grab were independently building custom model serving solutions, which created duplicated engineering effort and required data scientists to understand complex infrastructure instead of focusing on machine learning.

## Approach / What changed

Grab built Catwalk, a self-serve platform running TensorFlow Serving in containers on a managed Kubernetes cluster, allowing data scientists to deploy and update models simply by uploading exported model files to Amazon S3 buckets.

## Takeaways

- TensorFlow Serving continuously monitors the configured Amazon S3 model bucket to automatically load newly uploaded model versions without downtime.
- Kubernetes manages model workload isolation and auto-scaling by assigning TensorFlow Serving pods to cluster nodes with sufficient CPU and memory resources.
- Catwalk provisions Kubernetes Deployment, Service, and Ingress resources alongside an ingress controller to configure load balancing and expose HTTP endpoints for each model.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Deployment](https://yomu.fyi/topic/deployment), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/catwalk-serving-machine-learning-models-at-scale)