---
title: "Re-imagining ML Serving Infra: My Winter Internship at Ramp"
description: "Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing."
---

# Re-imagining ML Serving Infra: My Winter Internship at Ramp

[Ramp](https://yomu.fyi/company/ramp) · Michael Jiang · Apr 10, 2026

**Type:** Problem & solution

## Summary

Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.

## Context

ml-pipelines supported flexible machine-learning dependencies but redeployed models with app changes and did not always enforce strict versioning. Intelligence Platform required 99.9%+ uptime and very low latency, yet hosted models directly, limiting adoption of advanced models, creating dependency conflicts, and imposing resource and maintenance costs as usage grew.

## Approach / What changed

mw-serve introduced a dedicated serving layer that stores model weights in MLflow and a base image for the training environment in ECR. On demand, it retrieves both and serves the model in an isolated instance. The implementation added deployment safety checks, pre-traffic liveness checks, server-side monitors, detailed client and server observability, connection keepalive and startup warmups, and garbage-collection tuning.

## Takeaways

- Models are deployed only after checks confirm they can answer inference requests, and undeployment fails when load-balancer logs show production traffic in the previous two weeks.
- Datadog spans around inner TCP and TLS connections showed that connection churn caused most of the slowest requests; longer keepalives and startup warmups reduced those spikes.
- Processing-level observability linked spiky latency to memory drops and garbage-collection pauses blocking Python’s global interpreter lock; tuned thresholds and frozen objects removed more outliers.

**Tags:** [Model Serving](https://yomu.fyi/topic/model-serving), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Ramp](https://builders.ramp.com/post/re-imagining-ml-serving-infra)
- Source URL: https://builders.ramp.com/post/re-imagining-ml-serving-infra
- Ingested by Yomu: 2026-09-01T01:34:58.300Z

[Read original post](https://builders.ramp.com/post/re-imagining-ml-serving-infra)