Re-imagining ML Serving Infra: My Winter Internship at Ramp

Summary

Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.

Context

ml-pipelines supported flexible machine-learning dependencies but redeployed models with app changes and did not always enforce strict versioning. Intelligence Platform required 99.9%+ uptime and very low latency, yet hosted models directly, limiting adoption of advanced models, creating dependency conflicts, and imposing resource and maintenance costs as usage grew.

Approach / What changed

mw-serve introduced a dedicated serving layer that stores model weights in MLflow and a base image for the training environment in ECR. On demand, it retrieves both and serves the model in an isolated instance. The implementation added deployment safety checks, pre-traffic liveness checks, server-side monitors, detailed client and server observability, connection keepalive and startup warmups, and garbage-collection tuning.

Takeaways

• Models are deployed only after checks confirm they can answer inference requests, and undeployment fails when load-balancer logs show production traffic in the previous two weeks.
• Datadog spans around inner TCP and TLS connections showed that connection churn caused most of the slowest requests; longer keepalives and startup warmups reduced those spikes.
• Processing-level observability linked spiky latency to memory drops and garbage-collection pauses blocking Python’s global interpreter lock; tuned thresholds and frozen objects removed more outliers.

Related reading