Catwalk: Serving Machine Learning Models at Scale

Summary

As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.

Takeaways

1. 1

   TensorFlow Serving continuously monitors the configured Amazon S3 model bucket to automatically load newly uploaded model versions without downtime.

2. 2

   Kubernetes manages model workload isolation and auto-scaling by assigning TensorFlow Serving pods to cluster nodes with sufficient CPU and memory resources.

3. 3

   Catwalk provisions Kubernetes Deployment, Service, and Ingress resources alongside an ingress controller to configure load balancing and expose HTTP endpoints for each model.