Re-imagining ML Serving Infra: My Winter Internship at Ramp

Summary

Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.

Takeaways

1. 1

   Models are deployed only after checks confirm they can answer inference requests, and undeployment fails when load-balancer logs show production traffic in the previous two weeks.

2. 2

   Datadog spans around inner TCP and TLS connections showed that connection churn caused most of the slowest requests; longer keepalives and startup warmups reduced those spikes.

3. 3

   Processing-level observability linked spiky latency to memory drops and garbage-collection pauses blocking Python’s global interpreter lock; tuned thresholds and frozen objects removed more outliers.