Evolution of Catwalk: Model serving platform at Grab

Summary

Grab developed and scaled Catwalk, an internal machine learning model serving platform, to address operational bottlenecks, low resource utilization, and deployment friction between data scientists and backend engineers. The platform transitioned from an admin-managed TensorFlow Serving setup into a low-code self-service system supporting PyTorch and ONNX, before replacing complex Helm charts with Kubernetes Custom Resource Definitions for declarative, blue-green deployment orchestration. To support complex business workflows and multi-model applications, Grab subsequently introduced Catwalk Orchestrator with bundled deployments that allow individual services to scale independently. Across two years, the orchestrator architecture expanded to 200 deployed applications serving approximately 1,400 production machine learning models.

Context

Prior ad-hoc ML model serving at Grab caused operational overhead, low resource utilization of around 1%, duplicated engineering efforts, and friction between data scientists and backend teams.

Approach / What changed

Grab evolved Catwalk from a managed TensorFlow Serving platform into a self-service system, replaced Helm charts with Kubernetes Custom Resource Definitions for safer deployments, and built Catwalk Orchestrator with bundled Kubernetes deployments.

Takeaways

• Replacing Helm charts with Kubernetes Custom Resource Definitions enabled automated blue-green deployments, capacity management, and constraint validation through a custom state machine.
• Using bundled deployments allowed each model and orchestration component in an ML application to operate as a separate Kubernetes deployment with independent scaling.
• Adopting a high-code orchestrator alongside mandatory staging load testing expanded platform scale to 200 orchestrators serving 1,400 models.

Related reading

• Grab · Nov 29, 2024

  Supercharging LLM application development with LLM-Kit

  Grab developed LLM-Kit to resolve standardization, security, observability, and infrastructure provisioning bottlenecks across its generative AI development efforts. Submitting an initial application request automatically triggers project generation in GitLab, outputting a modular codebase along with Terraform definitions for Amazon EKS and ECR. The scaffolded architecture incorporates FastAPI, LangChain, OpenID Connect authentication helpers, and PGVector alongside HashiCorp Vault for credential management. Observability and quality tracking are built in through direct integrations with Datadog and LangSmith evaluations. By providing these standardized paved-road templates, the framework has onboarded hundreds of generative AI applications and saved teams an estimated 1.5 weeks of initial setup time.

  Boon Zhan Chew

• Grab · Dec 5, 2024

  How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication

  Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.

  Md Riyadh

• Grab · Sep 10, 2024

  Unveiling the process: The creation of our powerful campaign builder

  Grab details the event processing architecture behind Trident, its internal marketing campaign platform that evaluates If This, Then That (IFTTT) logic over Kafka streams. The core processing unit is a treatment consisting of an event, optional conditions, and actions. Complex campaign capabilities—such as counters, limits, and delays across multiple hours via recursive SQS message scheduling—are assembled from multiple coordinated treatments. To simplify campaign creation, Grab introduced a flowchart-like visual builder represented as a JSON node tree that compiles recursively into treatments while persisting node-to-treatment mappings to reconcile edits over time.

  Jie Zhang

• Grab · Nov 14, 2024

  Metasense V2: Enhancing, improving and productionisation of LLM powered data governance

  Grab scaled Metasense to automate metadata generation, column-level classification, and sensitivity tiering across its entire data lake. Post-rollout analysis revealed that high tag volumes, wide tables, and mixed content—such as nested JSON and customer communications—strained model capacity and led to missed Personally Identifiable Information (PII). To overcome these limitations, the team split the classification process into distinct PII and non-PII tasks, reduced prompt word counts, and partitioned tables with over 150 columns into smaller units. Integrating LangChain and LangSmith modernized the architecture by enabling direct prompt experimentation, custom metric tracking, and version-controlled deployments. The updated pipeline achieved low misclassification rates alongside automated alert thresholds to trigger model improvement protocols if errors increase.

  Nick Buhrer