Building a data plane from scratch: Stripe’s own high-performance distributed proxy

Summary

Stripe replaced Envoy in its service mesh after reliability, connection-cardinality, rollout-risk, and customization problems became increasingly difficult at its scale. Mesh-proxy, a Go proxy, implements HTTP/1.1 and HTTP/2, connects directly to Stripe’s service registry, and supports retries, hedging, rate limiting, and dynamic priority-based routing. Unlike Envoy’s worker-per-core model, it uses one read and one write goroutine per connection and multiplexes requests over a shared connection, reducing connection counts from one per worker to one per host. Load tests and deployment results showed about half the CPU under high load and roughly 50% lower latency, equivalent to around 3,000 days of cumulative inter-service latency saved daily. Direct registry integration also reduced unnecessary xDS updates, while custom Ruby balancing added 15–20% headroom before bursts overwhelm machines; Stripe is now pursuing anomaly detection and smart routing.

Context

Stripe’s service mesh handles millions of requests per second among thousands of services, and its reliability target is 99.9995%. As traffic grew, Envoy’s worker-based threading model created excessive upstream connection counts, updates through xDS used extra CPU and memory, upgrades became riskier, and its customization limits prevented some routing improvements.

Approach / What changed

Stripe built mesh-proxy, a Go replacement for Envoy with its own HTTP/1.1 and HTTP/2 implementations. It connects directly to the service registry, uses one read and one write goroutine per connection, multiplexes requests over shared connections, and supports retries, hedging, rate limiting, and dynamic priority-based routing.

Takeaways

• Mesh-proxy reduced high-load CPU usage by about half and cut latency by roughly 50% compared with Stripe’s Envoy deployment.
• Using one shared connection per host instead of one connection per worker reduced connection cardinality for fleetwide services and let them run reliably on smaller machines.
• Custom Ruby traffic balancing provided 15–20% more headroom before bursts overwhelmed backend machines, while direct service-registry integration reduced unnecessary xDS updates.

Related reading